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DOD  BUSINESS  TRANSFORMATION 

Improved  Management  Oversight  of  Business  System 
Modernization  Efforts  Needed 


Why  GAO  Did  This  Study 


What  GAO  Found 


The  Department  of  Defense  (DOD) 
invests  billions  of  dollars  annually  to 
modernize  its  business  systems, 
which  have  been  on  GAO’s  high-risk 
list  since  1995.  DOD  is  in  the  process 
of  implementing  nine  enterprise 
resource  planning  (ERP)  efforts 
which  perform  business-related  tasks 
such  as  general  ledger  accounting 
and  supply  chain  management.  These 
efforts  are  essential  to  transforming 
DOD’s  business  operations.  GAO  was 
asked  to  (1)  provide  the  status  of  the 
ERPs  as  of  December  31,  2009; 

(2)  determine  whether  selected  ERPs 
followed  schedule  and  cost  best 
practices;  and  (3)  determine  if  DOD 
has  defined  the  performance 
measures  to  assess  whether  the  ERPs 
will  meet  their  intended  business 
capabilities.  To  accomplish  these 
objectives,  GAO  reviewed  data  on  the 
status  of  each  ERP  from  the  program 
management  officers  and  interviewed 
the  DOD  and  military  departments’ 
chief  management  officers. 

What  GAO  Recommends 

In  addition  to  reiterating  its  existing 
recommendations,  GAO  is  making 
eight  recommendations  to  the 
Secretary  of  Defense  aimed  at 
improving  schedule  and  cost 
practices  and  the  development  of 
performance  measures  to  evaluate 
whether  the  ERPs’  intended  goals  are 
being  accomplished.  DOD  concurred 
with  our  recommendations  and  plans 
to  take  action  to  implement  them. 


Based  upon  the  data  provided  by  DOD,  six  of  the  nine  ERPs  have  experienced 
schedule  delays  ranging  from  2  to  12  years  and  five  have  incurred  cost 
increases  ranging  from  $530  million  to  $2.4  billion.  DOD  has  stated  that  the 
ERPs  will  replace  over  500  legacy  systems  that  cost  hundreds  of  millions  of 
dollars  to  operate  annually.  However,  delays  in  implementing  the  ERPs 
require  DOD  to  fund  the  legacy  systems  longer  than  anticipated,  thereby 
reducing  the  funds  available  for  other  DOD  priorities.  In  2007,  2008,  and  2009, 
GAO  made  19  recommendations  to  improve  the  management  of  DOD’s  ERP 
efforts.  While  DOD  agreed  with  the  recommendations,  14  have  not  yet  been 
fully  implemented. 

GAO  analyzed  four  of  the  nine  ERPs  to  determine  whether  scheduling  and 
cost  estimating  best  practices  were  being  followed.  Regarding  scheduling 
practices,  GAO  found  that  none  of  the  programs  had  developed  a  fully 
integrated  master  schedule  as  an  effective  tool  to  help  in  the  management  of 
the  programs.  A  reliable  schedule  is  crucial  to  estimating  the  overall  schedule 
and  cost  of  a  program.  Without  a  reliable  schedule,  DOD  is  unable  to  predict, 
with  any  degree  of  confidence,  if  the  estimated  completion  dates  are  realistic. 
Regarding  the  cost  estimates,  GAO  found  that  although  the  four  ERPs’  cost 
estimates  generally  met  the  criteria  for  three  of  the  four  best  practices — well- 
documented,  accurate,  and  comprehensive — three  ERPs  did  not  fully  meet  the 
credibility  criteria  because  potential  limitations  were  not  discussed.  More 
specifically,  the  three  ERPs  lacked  a  sensitivity  analysis  or  a  risk  and 
uncertainty  analysis  as  stipulated  in  GAO,  Office  of  Management  and  Budget, 
and  DOD  guidance,  thus  diminishing  the  credibility  of  the  estimates. 

While  the  ERPs  are  critical  to  transforming  DOD’s  business  operations, 
DOD  lacks  a  comprehensive  set  of  performance  measures  to  assess  these 
systems  and  their  contribution  to  transforming  business  operations. 
Management  needs  to  define  what  constitutes  a  successful  implementation  in 
terms  that  can  be  used  to  assess  whether  the  system  is  (1)  being  used  as 
expected  and  (2)  providing  the  intended  benefits.  Accordingly,  the  actual 
measures  used  to  accomplish  these  objectives  will  differ  depending  on  the 
system.  For  example,  measures  for  a  logistical  system  may  focus  on  reducing 
inventory  levels,  while  those  for  a  financial  system  may  focus  on  reducing 
prompt  payment  penalties.  Without  performance  measures  to  evaluate  how 
well  the  ERPs  are  accomplishing  their  intended  goals,  DOD  decision  makers 
do  not  have  all  the  information  they  need  to  determine  whether  DOD 
investments  are  accomplishing  their  desired  goals,  and  program  managers  do 
not  have  the  information  they  need  to  ensure  that  their  individual  program  is 
helping  DOD  to  achieve  business  transformation  and  thereby  improve  upon  its 
primary  mission  of  supporting  the  warfighter. 
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GAO 

^^Accountability  *  Integrity  *  Reliability 

United  States  Government  Accountability  Office 
Washington,  DC  20548 


October  7,  2010 
Congressional  Requesters 

The  Department  of  Defense’s  (DOD)  business  systems1  modernization 
program  has  been  on  our  high-risk  list2  since  1995  because  of  the  size, 
complexity,  and  significance  of  the  related  efforts.  DOD’s  business 
systems  modernization  entails  investments  in  and  the  implementation  of 
comprehensive,  integrated  business  systems  for  managing  an 
organization’s  resources,  commonly  referred  to  as  enterprise  resource 
planning  (ERP)3  systems  and  the  elimination  of  hundreds  of  legacy 
systems.  DOD  officials  have  said  that  successful  implementation  of  ERPs 
is  key  to  resolving  the  long-standing  weaknesses  in  the  department’s 
business  operations  in  areas  such  as  business  transformation,  financial 
management,  and  supply  chain  management,4  and  improving  the 
department’s  capability  to  provide  DOD  management  and  the  Congress 
with  accurate  and  reliable  information  on  the  results  of  its  operations. 

DOD  has  identified  10  ERPs,5 1  of  which  has  been  fully  implemented,  as 
essential  to  its  efforts  to  transform  its  business  operations.  According  to 
DOD,  as  of  December  2009,  it  had  invested  approximately  $5.8  billion  to 
develop  and  implement  these  ERPs  and  will  invest  additional  billions 
before  the  remaining  9  ERPs  are  fully  implemented.  Our  prior  reviews  of 
several  ERPs  have  found  that  the  department  has  not  effectively  employed 


DOD’s  business  systems  are  information  systems,  including  financial  and  nonfinancial 
systems  that  support  DOD  business  operations,  such  as  civilian  personnel,  finance,  health, 
logistics,  military  personnel,  procurement,  and  transportation. 

2GAO,  High-Risk  Series:  An  Update,  GAO-09-271  (Washington,  D.C.:  January  2009). 

3An  ERP  solution  is  an  automated  system  using  commercial  off-the-shelf  (COTS)  software 
consisting  of  multiple,  integrated  functional  modules  that  perform  a  variety  of  business- 
related  tasks  such  as  general  ledger  accounting,  payroll,  and  supply  chain  management. 

4These  areas  were  designated  as  high  risk  in  2005,  1995,  and  1990,  respectively. 

The  10  ERPs  are  as  follows:  Army — General  Fund  Enterprise  Business  System  (GFEBS), 
Global  Combat  Support  System-Army  (GCSS-Army),  and  Logistics  Modernization  Program 
(LMP);  Navy — Navy  Enterprise  Resource  Planning  (Navy  ERP)  and  Global  Combat  Support 
System-Marine  Corps  (GCSS-MC);  Air  Force — Defense  Enterprise  Accounting  and 
Management  System  (DEAMS)  and  Expeditionary  Combat  Support  System  (ECSS); 

Defense — Service  Specific  Integrated  Personnel  and  Pay  Systems  and  Defense  Agencies 
Initiative  (DAI);  and  Defense  Logistics  Agency — Business  System  Modernization  (BSM). 
According  to  DOD,  BSM  was  fully  implemented  in  July  2007. 
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acquisition  management  controls  or  delivered  the  promised  capabilities  on 
time  and  within  budget.6 

This  report  provides  information  to  support  your  continuing  oversight  of 
DOD’s  progress  in  modernizing  its  business  systems  to  address  long¬ 
standing  weaknesses  and  ultimately  to  transform  its  business  operations. 
As  agreed  with  your  office,  our  objectives  were  to  (1)  provide  the  status  as 
of  December  31,  2009  of  the  nine  ERPs  DOD  identified  as  essential  to 
transforming  its  business  operations,  (2)  assess  the  scheduling  and  cost 
estimating  practices  of  selected  ERPs  to  determine  the  extent  to  which  the 
program  management  offices  (PMO)  were  applying  best  practices,  and 
(3)  ascertain  whether  DOD  and  the  military  departments  have  defined  the 
performance  measures  to  determine  whether  the  systems  will  meet  their 
intended  business  capabilities. 

To  address  the  first  objective,  we  reviewed  status  information  obtained 
from  each  PMO,  such  as  the  reported  amount  of  funds  expended  on  the 
implementation  of  the  nine  ERPs,  the  estimated  number  of  legacy  systems 
to  be  replaced  by  each  ERP,  and  the  reported  annual  cost  of  maintaining 
these  legacy  systems.  We  also  reviewed  past  GAO  reports7  that  were 
specific  to  the  department’s  efforts  to  implement  the  nine  ERPs  to  identify 
prior  recommendations  and  assess  DOD’s  progress  in  addressing  the  19 
recommendations  discussed  in  these  reports. 

For  the  purposes  of  this  report,  we  did  not  include  information  on  the 
Defense  Logistics  Agency  (DLA)  Business  System  Modernization  (BSM)/ 
Enterprise  Business  System  (EBS).  According  to  DLA,  the  BSM  effort  was 
fully  implemented  in  July  2007,  and  transformed  how  the  agency  conducts 


6GAO,  Defense  Logistics:  Actions  Needed  to  Improve  Implementation  of  the  Army 
Logistics  Modernization  Program,  GAO-10-461  (Washington,  D.C.:  Apr.  30,  2010);  DOD 
Business  Systems  Modernization:  Navy  Implementing  a  Number  of  Key  Management 
Controls  on  Enterprise  Resource  Planning  System,  but  Improvements  Still  Needed, 
GAO-09-841  (Washington,  D.C.:  Sept.  15,  2009);  DOD  Business  Systems  Modernization: 
Important  Managemen  t  Controls  Being  Implemented  on  Major  Navy  Program,  but 
Improvements  Needed  in  Key  Areas,  GAO-08-896  (Washington,  D.C.:  Sept.  8,  2008);  DOD 
Business  Transformation:  Air  Force’s  Current  Approach  Increases  Risk  That  Asset 
Visibility  Goals  and  Transformation  Priorities  Will  Not  Be  Achieved,  GAO-08-866 
(Washington,  D.C.:  Aug.  8,  2008);  DOD  Business  Systems  Modernization:  Key  Marine 
Corps  System  Acquisition  Needs  to  Be  Better  Justified,  Defined,  and  Managed, 
GAO-08-822  (Washington,  D.C.:  July  28,  2008);  and  DOD  Business  Transformation:  Lack  of 
an  Integrated  Strategy  Puts  the  Army’s  Asset  Visibility  System  Investments  at  Risk, 
GAO-07-860  (Washington,  D.C.:  July  27,  2007). 

7GAO- 10-461,  GAO-09-841,  GAO-08-896,  GAO-08-866,  GAO-08-822,  and  GAO-07-860. 
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its  operations  in  five  core  business  processes:  order  fulfillment,  demand 
and  supply  planning,  procurement,  technical/quality  assurance,  and 
financial  management.  Subsequently,  in  September  2007,  the  name  of  the 
program  was  changed  to  the  EBS,  which  is  a  continuation  of  the  ERP’s 
capabilities  to  support  internal  agency  operations. 

To  address  the  second  objective,  we  assessed  the  scheduling  and  cost 
estimating  practices  for  four  of  the  nine  ERPs8  to  determine  the  extent  to 
which  the  PMOs  were  applying  best  practices  for  scheduling  and  cost 
estimating.  For  the  four  ERPs,  we  obtained  and  analyzed  the  most  current 
schedule  and  cost  estimate  for  each  program  and  compared  them  against 
the  criteria  set  forth  in  GAO’s  cost  guide.9  In  using  the  guide,  we 
determined  the  extent  to  which  the  schedule  was  prepared  in  accordance 
with  the  best  practices10  that  are  fundamental  to  having  a  reliable  schedule. 
In  assessing  each  program’s  cost  estimates,  we  used  the  GAO  cost  guide  to 
evaluate  the  PMOs’  estimating  methodologies,  assumptions,  and  results  to 
determine  whether  the  cost  estimates  were  comprehensive,  accurate,  well- 
documented,  and  credible.  We  did  not  conduct  detailed  schedule  and  cost 
assessments  for  the  remaining  five  programs  because  (1)  the 
implementation  strategy  has  not  been  fully  defined  for  two  of  the  ERPs, 

(2)  one  of  the  ERPs  is  near  full  deployment,  and  (3)  we  have  previously 
reported11  on  two  ERPs’  schedule  and  cost  estimating  practices. 

To  address  the  third  objective,  we  reviewed  the  extent  to  which  DOD  and 
the  military  departments  included  performance  measures  in  their 
congressional  reports  on  business  transformation.  In  addition,  we  met 
with  the  military  departments’  deputy  chief  management  officers  (DCMO) 
to  obtain  an  understanding  of  how  they  define  success  in  terms  of 
deploying  their  respective  ERPs.  We  also  met  with  the  DOD  DCMO  and 
the  Director  of  the  Business  Transformation  Agency  (BTA)  to  obtain  an 
understanding  of  their  respective  roles  and  responsibilities  in  the  oversight 
of  DOD’s  ERP  implementation  efforts.  Additional  details  on  our  scope  and 
methodology  are  presented  in  appendix  I. 


8We  reviewed  the  Army’s  GFEBS  and  GCSS-Army  and  the  Air  Force’s  DEAMS  and  ECSS. 

°GAO,  GAO  Cost  Estimating  and  Assessment  Guide  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009). 

10GAO-09-3SP. 

uGAO-08-822  and  GAO-08-896. 
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Background 


We  conducted  this  performance  audit  from  June  2009  through  October 
2010  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives.  We  requested  comments 
on  a  draft  of  this  report  from  the  Secretary  of  Defense  or  his  designee.  We 
received  written  comments  from  the  Deputy  Chief  Management  Officer, 
which  are  reprinted  in  appendix  II. 


DOD  is  one  of  the  largest  and  most  complex  organizations  in  the  world.  In 
fiscal  year  2009,  DOD  reported  that  its  operations  consisted  of  $1.8  trillion 
in  assets,  $2.2  trillion  in  liabilities,  approximately  3.2  million  military  and 
civilian  personnel — including  active  and  reserve  components — and 
disbursements  of  over  $947  billion.12  Execution  of  these  operations  spans  a 
wide  range  of  defense  organizations,  including  the  military  departments 
and  their  respective  major  commands  and  functional  activities,  large 
defense  agencies  and  field  activities,  and  various  combatant  and  joint 
operational  commands  that  are  responsible  for  military  operations  for 
specific  geographic  regions  or  theaters  of  operation.  To  execute  military 
operations,  the  department  performs  interrelated  and  interdependent 
business  functions,  including  financial  management,  logistics 
management,  health  care  management,  and  procurement.  To  support  its 
business  functions,  DOD  has  reported  that  it  relies  on  about  2,080  business 
systems,13  including  accounting,  acquisition,  logistics,  and  personnel 
systems. 


12The  reported  amounts  are  not  audited.  In  November  2009,  the  DOD  Inspector  General 
reported  that  because  of  long-standing  internal  control  weaknesses,  DOD’s  annual  financial 
statements,  which  included  these  reported  amounts,  were  not  accurate  and  reliable. 

13DOD  excludes  from  its  business  systems  those  designated  as  national  security  systems 
under  Section  2222  (j)  of  Title  10,  United  States  Code.  National  security  systems  are 
intelligence  systems,  cryptologic  activities  related  to  national  security,  military  command 
and  control  systems,  and  equipment  that  is  an  integral  part  of  a  weapon  or  weapons  system 
or  is  critical  to  the  direct  fulfillment  of  military  or  intelligence  missions. 
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Funding  of  DOD’s  Business 
Systems 


To  fund  its  existing  business  systems  environment,  DOD  requested  for 
fiscal  year  2011  nearly  $17.4  billion  to  operate,  maintain,  and  modernize  its 
reported  2,080  business  systems  (see  fig.  1).  Of  this  amount,  about 
$12.2  billion  is  for  operations  and  maintenance  and  the  remaining 
$5.2  billion  is  for  planned  or  ongoing  DOD  business  systems  development 
modernization  efforts. 
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Figure  1 :  DOD’s  Fiscal  Year  2011  Business  Systems  Budget  Request  by  DOD  Components  (Dollars  in  Thousands) 


Current  Development/ 

Component  services  modernization  Total  Percent  10% 


Army  $3,031,957  $1,768,150  $4,800,107 

|  27.7% 

Air  Force  $2,323,175  $1,666,103  $3,989,278 

|  23.0% 

Navy  $2,310,296  $536,492  $2,846,788 

|  16.4% 

TRICARE  Management  Activity  $1,403,434  $403,857  $1,807,291 

|  10.4% 

Defense  Logistics  Agency  $763,438  $160,478  $923,916 

|  5.3% 

Defense  Information  $689,584  $25,190  $714,774  1 

Systems  Agency  P 

|  4.1% 

Defense  Finance  and  $397,239  $29,812  $427,051  1 

Accounting  Service  t 

|  2.5% 

Defense  Human  $253,215  $67,950  $321,165  1 

Resources  Activity  t 

|  1.9% 

Transportation  Command  $186,370  $101,973  $288,343 

|  1.7% 

Business  Transformation  Agency  $53,332  $145,190  $198,522 

1 1.2% 

SeTvSie9100  HeadqUart6rS  $153,579  $27’119  $180’698  I 

|  1.0% 

Missile  Defense  Agency  $0  $152,208  $152,208 

|  0.8% 

Defense  Commissary  Agency  $117,861  $3,616  $121,477 

|  0.7% 

Defense  Contract  $103,391  $13,933  $117,324  1 

Management  Agency  P 

|  0.7% 

Department  of  Defense  $94,590  $0  $94,590  1 

Dependents  Education  P 

|  0.6% 

Office  of  the  Secretary  $29,755  $49,098  $78,853 

of  Defense  t 

|  0.5% 

Joint  Chiefs  of  Staff  $60,151  $10,963  $71,114 

0.4% 

Other  DOD  components  $180,479  $23,104  $203,583 

|  1.2% 

Total  $12,151,846  $5,185,236  $17,337,082  100% 


Source:  GAO  based  upon  fiscal  year  201 1  budget  request  data  provided  by  DOD.  This  data  has  not  been  validated. 


The  Office  of  Management  and  Budget  (OMB)  requires  that  funds 
requested  for  information  technology  (IT)  projects  be  classified  as  either 
“steady  state”  (or  “current  services”  in  DOD)  or  as 
“development/modernization.”  Current  services  represents  funds  for 
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operating  and  maintaining  systems  at  current  levels  (i.e.,  without  major 
enhancements).  The  development  modernization  budget  category 
represents  funds  for  developing  new  IT  systems  or  making  major 
enhancements  to  existing  systems.  Some  systems  have  both  current 
services  and  development  modernization  funding.  While  current  services 
are  to  be  used  for  operating  the  system  at  various  locations,  development 
modernization  funds  are  to  be  used  for  activities  such  as  developing  and 
expanding  system  functionality  at  existing  locations  and  deploying  the 
system  to  new  locations.  Generally,  current  services  are  financed  through 
Operation  and  Maintenance  appropriations,  whereas  development 
modernization  funding  can  come  from  several  or  a  combination  of  several 
appropriations,  such  as  Research,  Development,  Test,  and  Evaluation; 
Procurement;  or  the  Defense  Working  Capital  Fund. 


DOD’s  Acquisition  System 
Framework 


•  Materiel  solution  analysis  (previously  concept  refinement).  The 

purpose  of  this  phase  is  to  refine  the  initial  system  solution  (concept)  and 
create  a  strategy  for  acquiring  the  solution.  A  decision  is  made  at  the  end 
of  this  phase  (Milestone  A)  regarding  whether  to  move  to  the  next  phase. 

•  Milestone  A  authorizes  acquisition  of  the  program  and  permission  to 
begin  planning  and  development  of  the  system  technology. 

•  Technology  development.  The  purpose  of  this  phase  is  to  determine  the 
appropriate  set  of  technologies  to  be  integrated  into  the  investment 
solution  by  iteratively  assessing  the  viability  of  the  various  technologies 
while  simultaneously  refining  user  requirements.  Once  the  technology  has 
been  demonstrated,  a  decision  is  made  (Milestone  B)  whether  to  move  to 
the  next  phase. 

•  Milestone  B  authorizes  product  development  of  the  program  based 
on  well-defined  technology  and  a  reasonable  system  design  plan. 

•  Engineering  and  manufacturing  development  (previously  system 
development  and  demonstration).  The  purpose  of  this  phase  is  to 


ERPs  are  developed  within  the  defense  acquisition  system  framework, 
which  is  intended  to  translate  mission  needs  and  requirements  into  stable, 
affordable,  and  well-managed  acquisition  programs.14  The  defense 
acquisition  system  framework  was  updated  in  December  2008  and  consists 
of  five  program  life-cycle  phases  and  three  related  milestone  decision 
points  which  are  described  below. 


14DOD  Directive  5000.01,  The  Defense  Acquisition  System  (Nov.  20,  2007). 
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develop  a  system  and  demonstrate  through  developer  testing  that  the 
system  can  function  in  its  target  environment.  A  decision  is  made  at  the 
end  of  this  phase  (Milestone  C)  whether  to  move  to  the  next  phase. 

•  Milestone  C  authorizes  entry  of  the  system  into  the  production  and 
deployment  phase  or  into  limited  deployment  in  support  of  operational 
testing. 

•  Production  and  deployment.  The  purpose  of  this  phase  is  to  achieve  an 
operational  capability  that  satisfies  the  mission  needs,  as  verified  through 
independent  operational  test  and  evaluation,  and  to  implement  the  system 
at  all  applicable  locations. 

•  Operations  and  support.  The  purpose  of  this  phase  is  to  operationally 
sustain  the  system  in  the  most  cost-effective  manner  over  its  life  cycle. 


Overview  of  DOD  Business  In  2005,  DOD  adopted  a  “tiered  accountability”  approach  to  improve 
Systems  Investment  control  and  accountability  over  the  billions  of  dollars  it  invests  annually  in 

Review  Process  DOD  business  systems.  Under  this  approach,  executive  leadership  for  the 

direction,  oversight,  and  execution  of  DOD  investments  is  the 
responsibility  of  several  entities  within  DOD  and  its  components.  As 
indicated  below,  the  investment  control  process  begins  at  the  component 
level  and  works  its  way  up  through  a  hierarchy  of  review  and  approval 
authorities,  depending  on  the  size  and  significance  of  the  investment.16 

•  Defense  Business  Systems  Management  Committee  (DBSMC)  serves  as 
the  highest-ranking  governance  body  for  business  systems  modernization 
activities  and  approves  funding  request  for  investments  costing  more  than 
$1  million  within  the  department. 


15There  are  five  tiers  of  business  systems.  Tier  1  systems  include  all  large,  expensive  system 
programs  classified  as  a  major  automated  information  system  (MAIS)  or  a  major  defense 
acquisition  program  (MDAP)  and  subject  to  the  most  extensive  statutory  and  regulatory 
reporting  requirements.  Tier  2  systems  include  those  with  modernization  efforts  of 
$10  million  or  greater  but  that  are  not  designated  as  MAIS  or  MDAP  or  programs  that  have 
been  designated  as  investment  review  board  programs  of  interest  because  of  their  effect  on 
DOD  transformation  objectives.  Tier  3  systems  include  those  with  modernization  efforts 
that  have  anticipated  costs  greater  than  $1  million  but  less  than  $10  million.  Tier  4  includes 
systems  with  development/modemization  cost  of  $1  million  or  less.  Tier  5  includes  systems 
in  operation  and  maintenance  or  sustainment. 
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•  Investment  review  boards  (IRB)16  are  responsible  for  the  review,  approval, 
and  oversight  of  the  planning,  design,  acquisition,  deployment,  operation, 
maintenance,  and  modernization  of  defense  business  systems.  The  IRBs 
are  also  responsible  for  recommending  business  systems  to  the  DBSMC 
for  certification,17  which  equates  to  recommending  funding,  for  all 
business  system  investments  costing  more  than  $1  million. 

•  The  Milestone  Decision  Authority  (MDA)  is  the  senior  DOD  official  who 
has  overall  authority  to  approve  entry  of  an  acquisition  program  into  the 
next  phase  of  the  acquisition  process  and  is  accountable  for  cost, 
schedule,  and  performance  reporting,  including  congressional  reporting. 

•  DOD  Component  Acquisition  Executive  is  responsible  for  providing  a 
written  memorandum  to  the  MDA  through  the  cognizant  IRB  that 

(1)  states  that  the  program  complies  with  applicable  DOD  statutory  and 
regulatory  requirements,  (2)  describes  any  conditions  or  issues  applicable 
to  the  requested  acquisition  decision,  and  (3)  recommends  approval  of  the 
acquisition  decision  request. 


16The  five  IRBs  are  (1)  financial  management  established  by  the  Under  Secretary  of 
Defense  (Comptroller);  (2)  weapon  systems  life-cycle  management  and  materiel  supply  and 
services  management  established  by  the  Under  Secretary  of  Defense  (Acquisition, 
Technology  and  Logistics);  (3)  real  property  and  installations  life-cycle  management 
established  by  the  Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics); 

(4)  human  resources  management  established  by  the  Under  Secretary  of  Defense  for 
Personnel  and  Readiness;  and  (5)  Department  of  Defense  Chief  Information  Officer 
established  by  the  Assistant  Secretary  of  Defense  (Networks  and  Information 
Integration)/DOD  Chief  Information  Officer. 

^Ronald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year  2005,  Pub.  L.  No. 
108-375,  §  332, 118  Stat.  1811,  1851-1856  (Oct.  28,  2004),  codified  in  part  at  10  U.S.C.  § 

2222,  directs  that  DOD  may  not  obligate  appropriated  funds  for  a  defense  business  system 
modernization  with  a  total  cost  of  more  than  $1  million  unless,  the  approval  authority — 
that  is  the  appropriate  IRB — certifies  that  the  business  system  modernization  either 
(1)  complies  with  the  department’s  business  enterprise  architecture,  (2)  is  necessary  to 
achieve  a  critical  national  security  capability  or  address  a  critical  requirement  in  an  area 
such  as  safety  or  security,  or  (3)  is  necessary  to  prevent  a  significant  adverse  effect  on  an 
essential  project  in  consideration  of  alternative  solutions.  This  certification  must  also  be 
approved  by  the  DBSMC.  Also,  as  of  October  28,  2009,  the  fiscal  year  2010  National  Defense 
Authorization  Act,  Pub.  L.  No.  111-84,  §1072,  123  Stat.  2190,  2470  (Oct.  28,  2009),  amended 
this  requirement.  This  amendment  requires  the  chief  management  officer  of  the  military 
services,  or  for  defense  agencies,  the  DOD  DCMO,  to  assess  whether  (1)  the  business 
process  that  the  system  supports  will  be  as  streamlined  and  efficient  as  possible  and  (2)  the 
need  to  tailor  commercial-off-the-shelf  systems  to  meet  unique  requirements  or  incorporate 
unique  interfaces  has  been  eliminated  or  reduced  to  the  maximum  extent  practicable.  This 
assessment  is  required  both  as  a  precondition  of  the  approval  of  any  new  business  system 
modernization  with  a  cost  over  $1  million,  and  as  a  review  of  any  previously  approved 
business  system  modernization  with  a  cost  over  $100  million. 
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•  A  DOD  component  pre-certification  authority  (PCA)  acts  as  the 
component’s  principle  point  of  contact  with  the  IRBs.  The  PCA  is 
responsible  for  identifying  the  component’s  systems  that  require  IRB 
certifications  and  prepares,  reviews,  approves,  and  validates  investment 
documentation  as  required.  The  PCA  also  submits  to  the  appropriate  IRB 
the  component’s  precertification  memorandum  that  asserts  the  status  and 
validity  of  the  business  system’s  investment  information  during  the 
certification  and  annual  review  processes. 

The  MDA,  IRBs,  the  DBSMC  or  a  combination  of  these  can  place 
conditions  or  issues  needing  resolution  upon  the  individual  programs 
during  the  defense  business  system’s  funding  certification  and  acquisition 
decision  review  processes.  These  conditions  are  generally  noted  in  a 
memorandum.  Further,  DOD’s  business  investment  management  system 
includes  two  types  of  reviews  for  business  systems:  certification  and 
annual  reviews.  Certification  reviews  apply  to  modernization  projects  with 
total  costs  over  $1  million.  These  reviews  focus  on  program  alignment 
with  the  business  enterprise  architecture  and  must  be  completed  before 
components  obligate  funds  for  programs.  As  noted  above,  the  IRBs 
recommend  certification  to  the  DBSMC,  which  approves  the  expenditure 
of  funds.  The  annual  reviews  apply  to  all  business  programs  and  are 
undertaken  to  determine  whether  the  system  development  effort  is 
meeting  its  milestones  and  addressing  its  certification  conditions. 

Additionally,  the  Duncan  Hunter  National  Defense  Authorization  Act  for 
Fiscal  Year  2009  directs  that  the  executive-level  oversight  of  DOD-wide 
business  systems  modernization  and  overall  business  transformation — 
including  defining  and  measuring  success  in  enterprise  resource 
planning — is  the  responsibility  of  a  military  department-level  chief 
management  officer  and  the  DCMO.18 


DOD’s  ERP  Efforts  The  department  stated  that  the  following  nine  ERPs  are  critical  to 

transforming  the  department’s  business  operations  and  addressing  some 
of  its  long-standing  weaknesses.  A  brief  description  of  each  ERP  is 
presented  below. 

•  The  General  Fund  Enterprise  Business  System  (GFEBS)  is  intended  to 
support  the  Army’s  standardized  financial  management  and  accounting 


18Pub.  L.  No.  110-417,  div.  A,  title  IX;  §908,  122  Stat.  4356,  4569  (Oct.  14,  2008). 
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practices  for  the  Army’s  general  fund,19  with  the  exception  of  that  related 
to  the  Army  Corps  of  Engineers,  which  will  continue  to  use  its  existing 
financial  system,  the  Corps  of  Engineers  Financial  Management  System.20 
GFEBS  will  allow  the  Army  to  share  financial,  asset  and  accounting  data 
across  the  active  Army,  the  Army  National  Guard,  and  the  Army  Reserve. 
The  Army  estimates  that  when  fully  implemented,  GFEBS  will  be  used  to 
control  and  account  for  about  $140  billion  in  spending. 

•  The  Global  Combat  Support  System-Army  (GCSS-Army)  is  expected  to 
integrate  multiple  logistics  functions  by  replacing  numerous  legacy 
systems  and  interfaces.  The  system  will  provide  tactical  units  with  a 
common  authoritative  source  for  financial  and  related  non-financial  data, 
such  as  information  related  to  maintenance  and  transportation  of 
equipment.  The  system  is  also  intended  to  provide  asset  visibility  for 
accountable  items.  GCSS-Army  will  manage  over  $49  billion  in  annual 
spending  by  the  active  Army,  National  Guard,  and  the  Army  Reserve. 

•  The  Logistics  Modernization  Program  (LMP)  is  intended  to  provide  order 
fulfillment,  demand  and  supply  planning,  procurement,  asset  management, 
material  maintenance,  and  financial  management  capabilities  for  the 
Army’s  working  capital  fund.  The  Army  has  estimated  that  LMP  will  be 
populated  with  6  million  Army-managed  inventory  items  valued  at  about 
$40  billion  when  it  is  fully  implemented. 

•  The  Navy  Enterprise  Resource  Planning  System  (Navy  ERP)  is  intended  to 
standardize  the  acquisition,  financial,  program  management,  maintenance, 
plant  and  wholesale  supply,  and  workforce  management  capabilities  at  six 
Navy  commands.21  Once  it  is  fully  deployed,  the  Navy  estimates  that  the 
system  will  control  and  account  for  approximately  $71  billion,  or  50 
percent,  of  the  Navy’s  estimated  appropriated  funds — after  excluding  the 
appropriated  funds  for  the  Marine  Corps  and  military  personnel  and  pay. 


19The  general  fund  can  be  defined  as  the  fund  into  which  receipts  are  deposited,  except 
those  from  specific  sources  required  by  law  to  be  deposited  into  other  designated  funds 
and  from  which  appropriations  are  made  by  Congress  to  carry  on  the  general  and  ordinary 
operations  of  the  government. 

"According  to  the  GFEBS  PMO,  once  the  system  is  fully  operational  the  Army  will  assess 
the  feasibility  of  GFEBS  becoming  the  system  of  record  for  the  Corps  of  Engineers. 

21The  six  Navy  commands  are  the  Naval  Air  Systems  Command,  the  Naval  Supply  Systems 
Command,  the  Space  and  Naval  Warfare  Systems  Command,  the  Naval  Sea  Systems 
Command,  the  Strategic  Systems  Program,  and  the  Office  of  Naval  Research  and  Strategic 
Systems  Planning. 
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•  The  Global  Combat  Support  System-Marine  Corps  (GCSS-MC)  is  intended 
to  provide  the  deployed  warfighter  enhanced  capabilities  in  the  areas  of 
warehousing,  distribution,  logistical  planning,  depot  maintenance,  and 
improved  asset  visibility.  According  to  the  PMO,  once  the  system  is  fully 
implemented,  it  will  control  and  account  for  approximately  $1.2  billion  of 
inventory. 

•  The  Defense  Enterprise  Accounting  and  Management  System  (DEAMS)  is 
intended  to  provide  the  Air  Force  the  entire  spectrum  of  financial 
management  capabilities,  including  collections,  commitments  and 
obligations,  cost  accounting,  general  ledger,  funds  control,  receipts  and 
acceptance,  accounts  payable  and  disbursement,  billing,  and  financial 
reporting  for  the  general  fund.  According  to  Air  Force  officials,  when 
DEAMS  is  fully  operational,  it  is  expected  to  maintain  control  and 
accountability  for  about  $160  billion. 

•  The  Expeditionary  Combat  Support  System  (ECSS)  is  intended  to  provide 
the  Air  Force  a  single,  integrated  logistics  system — including 
transportation,  supply,  maintenance  and  repair,  engineering  and 
acquisition — for  both  the  Air  Force’s  general  and  working  capital  funds. 
Additionally,  ECSS  is  intended  to  provide  the  financial  management  and 
accounting  functions  for  the  Air  Force’s  working  capital  fund  operations. 
When  fully  implemented,  ECSS  is  expected  to  control  and  account  for 
about  $36  billion  of  inventory. 

•  The  Service  Specific  Integrated  Personnel  and  Pay  Systems  are  intended  to 
provide  the  military  departments  an  integrated  personnel  and  pay  system.22 

•  Defense  Agencies  Initiative  (DAI)  is  intended  to  modernize  the  defense 
agencies’  financial  management  processes  by  streamlining  financial 
management  capabilities  and  transforming  the  budget,  finance,  and 
accounting  operations.  When  DAI  is  fully  implemented,  it  is  expected  to 
have  the  capability  to  control  and  account  for  all  appropriated,  working 
capital  and  revolving  funds  at  the  defense  agencies  implementing  the 
system. 


22The  military  services  integrated  personnel  and  pay  system  is  a  replacement  for  the 
Defense  Integrated  Military  Human  Resources  System  that  was  intended  to  provide  a  joint, 
integrated,  standardized  personnel  and  pay  system  for  all  military  personnel. 
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Status  of  DOD’s  ERP 

Implementation 

Efforts 


Based  upon  the  information  provided  by  the  PMOs,  six  of  the  ERPs  have 
experienced  schedule  slippages  (see  table  1)  based  on  comparing  the 
estimated  date  that  each  program  was  originally  scheduled  to  achieve  full 
deployment23  to  the  full  deployment  date  as  of  December  2009.  For  the 
remaining  three  ERPs,  the  full  deployment  date  has  either  remained 
unchanged  or  has  not  been  established.  The  GFEBS  PMO  noted  that  the 
acquisition  program  baseline  approved  in  November  2008,  established  a 
full  deployment  date  in  fiscal  year  2011  and  that  date  remains  unchanged. 
Additionally,  according  to  the  GCSS-Army  PMO  a  full  deployment  date  has 
not  been  established  for  this  effort.  The  PMO  noted  that  a  full  deployment 
date  will  not  be  established  for  the  program  until  a  full  deployment 
decision  has  been  approved  by  the  department.  A  specific  timeframe  has 
not  been  established  for  when  the  decision  will  be  made.  Further,  in  the 
case  of  DAI,  the  original  full  deployment  date  was  scheduled  for  fiscal  year 
2012,  but  the  PMO  is  in  the  process  of  reevaluating  the  date  and  a  new  date 
has  not  yet  been  established. 


Table  1 :  Reported  Full  Deployment  Schedule  Slippage  for  Each  ERP  as  of 

December  31,  2009 

Originally  scheduled 

Actual  or  latest 

Component/system 

fiscal  year  for  full 

estimated  fiscal  year 

Schedule 

name 

deployment 

for  full  deployment 

slippage 

Army 

GFEBS 

2011 

2011 

None 

GCSS-Army 

a 

a 

Not  applicable 

LMP 

2005 

2011 

6  years 

Navy 

Navy  ERP 

2011 

2013 

2  years 

GCSS-MC 

2010 

2013 

3  years3 

Air  Force 

DEAMS 

2014 

2017 

3  years 

ECSS 

2012 

2016 

4  years 

23Full  deployment  means  with  respect  to  a  major  automated  information  system  program, 
the  fielding  of  an  increment  of  the  program  in  accordance  with  the  terms  of  a  full 
deployment  decision — the  final  decision  made  by  the  MDA  authorizing  an  increment  of  the 
program  to  deploy  software  for  operational  use.  Pub.  L.  No.  111-84,  div.  A,  §841,  123  Stat. 
2190,  2418  (Oct.  28,  2009),  the  National  Defense  Authorization  Act  for  Fiscal  Year  2010, 
directed  that  the  terminology  be  changed  from  full  operational  capability  to  full 
deployment. 
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Component/system 

name 

Originally  scheduled 
fiscal  year  for  full 
deployment 

Actual  or  latest 
estimated  fiscal  year 
for  full  deployment 

Schedule 

slippage 

DOD 

Service  Specific 
Integrated  Personnel 
and  Pay  Systems 

2006 

Army— 201 4 

Navy— 2017 

Air  Force — 2018 

1 2  years0 

DAI 

2012 

d 

Not  applicable 

Source:  DOD  program  management  offices. 


aThe  PMO  has  not  yet  determined  the  full  deployment  date. 

“The  PMO  stated  that  the  estimated  full  deployment  date  is  only  for  phase  1 .  The  full  deployment  date 
for  the  entire  program  has  not  yet  been  determined. 

“Originally,  this  ERP  was  referred  to  as  the  Defense  Integrated  Military  Human  Resources  System 
(DIMHRS)  and  was  intended  to  provide  a  joint,  integrated,  standardized  personnel/pay  system  for  all 
military  personnel  departmentwide.  The  original  full  deployment  date  represents  the  estimated  date 
for  DIMHRS.  Each  military  service  is  now  responsible  for  developing  its  own  integrated  personnel  and 
pay  system. 

dAs  of  December  2009,  the  DAI  PMO  had  not  determined  the  revised  full  deployment  date. 


Besides  schedule  slippages,  five  of  the  ERP  efforts  have  reported  a  cost 
increase  and  one  program — GFEBS — reported  a  cost  decrease  of 
$17  million  (see  table  2).  The  reported  life-cycle24  cost  estimate  for 
GCSS-MC  only  represents  the  estimated  cost  for  phase25 1  of  the  program. 
The  cost  of  the  remaining  phases  has  not  yet  been  determined  and 
therefore,  a  total  life-cycle  cost  estimate  for  the  entire  program  has  not 
been  determined.  Additionally,  a  current  life-cycle  cost  estimate  has  not 
been  determined  for  the  Service  Specific  Integrated  Personnel  and  Pay 
Systems  and  DAI. 


24A  life-cycle  cost  estimate  provides  an  accounting  of  all  resources  and  associated  cost 
elements  required  to  develop,  produce,  deploy,  and  sustain  a  particular  program.  The 
life-cycle  cost  estimate  encompasses  all  past,  present,  and  future  costs  for  every  aspect  of 
the  program,  regardless  of  funding  source. 

25ERPs  are  developed  in  accordance  with  various  models  using  terminology  that  varies 
among  defense  organizations  and  in  some  cases  even  within  a  given  military  service.  For 
example,  the  Army’s  GFEBS  refers  to  a  scheduled  segment  as  a  “release,”  and  within  a 
release,  there  are  “waves.”  The  Air  Force’s  DEAMS  program  refers  to  scheduled  segments 
as  “increments”  and  within  increments,  there  are  “spirals.”  For  the  purposes  of  this  report, 
we  refer  generally  to  scheduled  segments  of  implementation  as  “phases.” 
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Table  2:  Reported  Original  and  Current  Life-Cycle  Cost  Estimate  for  Each  ERP  as  of 


December  31,  2009 

Dollars  in  millions 

Component/system 

name 

Original  life-cycle 
cost  estimate 

Current  life-cycle 
cost  estimate 

Reported  cost 
increase 

Army 

GFEBS 

$1,354 

$1,337 

$(17) 

GCSS-Army 

$3,900 

$3,900 

0 

LMP 

$2,630 

$2,630a 

0 

Navy 

Navy  ERP 

$1,870 

$2,400 

$530 

GCSS-MC 

$126 

$934 

$808“ 

Air  Force 

DEAMS 

$1,100 

$2,048 

$948 

ECSS 

$3,000 

$5,200 

$2,200“ 

DOD 

Service  Specific 
Integrated  Personnel 
and  Pay  Systems 

$577“ 

Armyd 

Navy-$1,300 

Air  Force-$1 ,700 

At  least  $2,423 

DAI 

$209 

e 

Not  applicable 

Source:  DOD  Program  Management  Offices. 

“At  the  time  LMP  was  designated  as  a  major  automated  information  system  (MAIS)  program  in 
December  2007,  it  was  required  to  comply  with  the  DOD  guidance  for  MAIS  programs.  This  guidance 
requires,  among  other  things,  that  a  MAIS  program  have  a  completed  and  approved  acquisition 
program  baseline — the  baseline  description  of  the  program,  including  the  life-cycle  cost  estimate — 
prior  to  Milestone  B  approval.  The  $2.6  billion  is  the  only  life-cycle  cost  estimate  that  has  been 
developed  for  the  program. 

bThe  current  life-cycle  cost  estimate  for  GCSS-MC  is  for  phase  one.  The  remaining  two  phases  will 
have  separate  baselines. 

“Originally,  ECSS  was  to  be  implemented  in  three  phases,  but  now,  it  will  be  implemented  in  four 
phases. 

“The  original  life-cycle  cost  estimate  represents  the  estimate  for  DIMHRS.  While  the  Navy  and  Air 
Force  have  estimated  their  respective  life-cycle  cost  estimate,  the  Army  is  in  the  process  of 
completing  its  life-cycle  cost  estimate. 

“As  of  December  2009,  the  life-cycle  cost  estimate  for  DAI  had  not  been  finalized.  According  to  the 
PMO,  the  life-cycle  cost  estimate  is  expected  to  be  approved  at  Milestone  B  in  fiscal  year  201 1 . 


According  to  the  PMOs,  while  there  have  been  schedule  slippages  and  cost 
increases,  for  several  of  the  nine  ERP  efforts,  the  functionality  that  was 
envisioned  and  planned  when  each  program  was  initiated  remains  the 
same  today.  While  the  original  intent  of  each  program  remains  the  same, 
the  anticipated  savings  that  were  to  accrue  to  the  department  may  not  be 
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fully  realized.  Delays  in  implementing  the  ERPs  result  in  DOD  having  to 
fund  the  operation  and  maintenance  of  the  legacy  systems  longer  than 
anticipated,  thereby  reducing  funds  that  could  be  used  for  other  DOD 
priorities. 

Furthermore,  we  have  previously  reported  on  the  department’s  effort  in 
implementing  some  of  the  ERPs  and  made  19  recommendations  to 
improve  DOD’s  management  and  oversight  of  these  efforts.  As  of  October 
2010,  the  department  has  taken  sufficient  action  to  implement  5  of  the 
recommendations.  Appendix  III  provides  details  on  the  specific 
recommendations  and  the  department’s  efforts  to  address  them.  The 
following  information  describes  in  more  detail  the  status  of  each  ERP. 


General  Fund  Enterprise 
Business  System 


DOD  Program  Data  for  GFEBS,  as  of  December  31 , 2009 

Date  of  initiation:  October  2004 

Program  owner:  Assistant  Secretary  of  the  Army  for  Financial  Management  and 

Comptroller 

Reported  life-cycle  cost  estimate: 

$1,336.7  billion 

•  Development  and  Modernization 

$  642.4  million 

•  Operations  and  Maintenance 

$  694.3  million 

Reported  amount  expended:  $416.8  million 

Reported  legacy  systems  to  be  replaced:  87 

Reported  annual  cost  of  maintaining  legacy 

systems:  $57.8  million 

Number  of  system  interfaces:  56 

Date  of  last  certification  of  funding:  September  2,  2009,  by  the  DBSMC 

Number  of  system  users:  79,000 

Number  of  locations:  200 

Source:  DOD’s  GFEBS  Program  Management  Office.  These  data  have  not  been  validated. 


Program  Status  According  to  the  GFEBS  PMO,  the  system  will  be  implemented  in  four 

phases.  Phases  1  and  2  were  completed  in  October  2008  and  provided  full 
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functionality  to  250  users  at  the  Management  Command,  Fort  Jackson, 
South  Carolina.  The  implementation  of  phase  2  set  the  stage  for  GFEBS  to 
be  deployed  to  the  rest  of  the  Army.  The  PMO  currently  estimates  that 
phases  3  and  4  will  be  deployed  Army-wide  with  full  functionality  by 
December  2011.  PMO  officials  told  us  that  the  establishment  of  the 
December  2011  milestone  resulted  from  conditions  placed  on  the  GFEBS 
program  at  Milestone  B,  directing  the  Army  to  develop  an  integrated 
strategy  for  the  implementation  of  GFEBS  and  GCSS-Army — meaning  that 
both  systems  were  to  be  implemented  using  a  standard  configuration  and 
set  of  common  master  data.26  The  PMO  also  stated  that  the  original  life- 
cycle  cost  estimate  of  approximately  $1.3  billion  covering  fiscal  years  2005 
through  2022  remained  unchanged  as  of  December  31,  2009. 

On  May  30,  2009,  GFEBS  was  authorized  by  the  MDA  to  proceed  with  a 
limited  deployment  to  initial  operational  test  and  evaluation  (IOT&E)27 
sites.  In  January  2010  and  again  in  March  2010,  the  GFEBS  program  was 
authorized  to  continue  its  deployment  to  a  limited  number  of  sites. 
According  to  the  MDA,  this  limited  deployment  process  allows  the 
program  to  gain  additional  operational  experience  with  the  GFEBS 
application  and  conduct  additional  user  testing.  MDA  approval  is  required 
for  deployment  to  additional  sites  and  full  deployment.  Before  GFEBS  will 
be  granted  approval  for  full  deployment  of  phase  4,  the  PMO  must  address 
several  conditions28  that  were  placed  on  the  program  by  the  MDA. 
According  to  the  PMO,  all  of  the  conditions  were  addressed  in  December 
2009  and  presented  to  the  IRB  for  approval.  However,  the  decision  on  the 
deployment  of  the  system  to  additional  locations  is  pending  and  scheduled 
to  occur  during  fiscal  year  2010. 


26Master  data  are  that  persistent,  non-transactional  data  that  defines  a  business  entity  for 
which  there  is,  or  should  be,  an  agreed  upon  view  across  the  organization.  This  key 
business  information  may  include  data  about  customers,  products,  employees,  materials, 
and  suppliers.  Master  data  are  often  used  by  several  functional  groups  and  stored  in 
different  data  systems  across  an  organization  and  may  or  may  not  be  referenced  centrally; 
therefore,  the  possibility  exists  for  duplicate  master  data  and/or  inaccurate  master  data. 

“'IOT&E  are  conducted  on  production  or  production-representative  articles  to  determine 
whether  systems  are  operationally  effective  and  suitable. 

28Conditions  or  issues  needing  resolution  may  be  placed  upon  the  ERPs  by  the  MDA,  the 
IRB,  or  the  DBSMC  during  the  business  system’s  funding  certification  and  acquisition 
decision  review  milestone  process.  These  conditions  are  generally  noted  in  a 
memorandum. 
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In  December  2009,  the  U.S.  Army  Test  and  Evaluation  Command  (ATEC) 
reported  on  concerns  with  GFEBS’s  data  accuracy,  reliability,  and 
timeliness.29  More  specifically,  the  report  noted  that  Army  “installations 
certifying  year-end  data  with  caveats  and  notes  related  to  inaccurate, 
incomplete,  and  missing  data.”  Furthermore,  the  report  noted  that 
“because  of  incomplete  or  not  implemented  business  processes,  users  at 
times,  executed  their  mission  using  the  “workarounds”  of  the  legacy 
systems  that  the  GFEBS  is  intended  to  replace  or  subsume.”  The  report 
recommended  that  the  deployment  of  GFEBS  be  limited  until  the 
problems  are  resolved  and  the  corrective  actions  have  been  validated  by 
ATEC.  According  to  the  PMO,  in  conjunction  with  ATEC,  a  plan  of  action 
and  milestones  has  been  developed  to  address  the  issues.  The  PMO  noted 
that  GFEBS  is  undergoing  an  additional  operational  test  and  evaluation 
limited  user  test;  and  at  the  conclusion  of  the  testing,  a  determination  will 
be  made  whether  the  ATEC  issues  have  been  addressed. 


29U.S.  Army  Test  and  Evaluation  Command,  Operational  Test  Agency  Evaluation  Report 
for  the  General  Fund  Enterprise  Business  System.  (Alexandria,  Va.:  Dec.  16,  2009). 
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Global  Combat  Support 
System-Army 


Program  Status 


DOD  Program  Data  for  GCSS-Army,  as  of  December  31, 2009 

Date  of  initiation:  December  2003a 

Program  owner:  Army  Deputy  Chief  of  Staff  for  Logistics 

Reported  life-cycle  cost  estimate: 

$3.9  billion 

•  Development  and  Modernization 

$1 .8  billion 

•  Operations  and  Maintenance 

$2.1  billion 

Reported  amount  expended:  $581  million 

Reported  legacy  systems  to  be  replaced:  7 

Reported  annual  cost  of  maintaining  legacy  systems:  $63  million 

Number  of  system  interfaces:  106 

Date  of  last  certification  of  funding:  September  2,  2009,  by  the  DBSMC 

Number  of  system  users:  169,880 

Number  of  locations:  379 

Source:  DOD's  GCSS-Army  Program  Management  Office.  These  data  have  not  been  validated. 

aPrior  to  the  initiation  of  the  current  ERP  effort,  the  Army  had  been  developing  custom  software  since 
May  1997. 


GCSS-Army  is  being  implemented  in  three  phases  with  phases  1  and  2 
being  proof-of-concept  demonstrations  that  have  been  ongoing  since 
December  2007  at  the  National  Training  Center  (NTC)  in  Fort  Irwin, 
California  and  testing  and  evaluation  are  scheduled  to  be  completed  in 
January  2012.  The  GCSS-Army  team  is  conducting  critical  activities,  such 
as  data  cleansing  and  training  users  at  the  NTC  site.  Phase  3  is  intended  to 
provide  full  functionality  and  is  scheduled  to  begin  implementation  in 
October  2013,  but  a  full  deployment  date  has  not  yet  been  determined. 
According  to  the  PMO,  the  exact  locations  for  the  implementation  of  phase 
3  have  not  been  determined  because  the  deployment  schedule  by  specific 
location  has  not  yet  been  finalized. 

In  July  2008,  the  MDA,  in  approving  Milestone  B,  directed  GCSS-Army  to 
develop  and  implement  a  strategy  to  better  facilitate  interactions  with 
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GFEBS  and  LMP.  Under  the  federated  strategy,  GCSS-Army  will  use 
GFEBS’  financial  template  to  allow  the  Army  to  integrate  data  on  logistics, 
financial,  maintenance,  property,  and  accountability  of  assets.  This 
strategy  is  intended  to  standardize  transactional  input  and  business 
processes  across  the  Army  ERPs  to  enable  common  cost  management 
activities;  provide  accurate,  reliable,  and  real-time  data;  and  tie  budgets  to 
execution.  According  to  the  PMO,  this  change  in  implementation  strategy 
resulted  in 

•  the  Cost  Analysis  Improvement  Group’s  direction  that  an  additional  year 
of  support  be  added  to  the  cost  estimate  because  of  the  additional  time 
needed  to  deploy  the  system  and 

•  a  revised  strategy  that  resulted  in  an  increase  in  the  number  of  required 
reports,  interfaces,  conversions,  and  extensions  that  need  to  be  developed 
or  tested  for  GCSS-Army’s  integration  with  GFEBS. 


Logistics  Modernization 
Program 


DOD  Program  Data  for  LMP,  as  of  December  31 , 2009 
Date  of  initiation:  December  1999 
Program  owner:  Army  Materiel  Command 

Reported  life-cycle  cost  estimate: 

•  Development  and  Modernization 

•  Operations  and  Maintenance 

Reported  amount  expended:  $1.1  billion 
Reported  legacy  systems  to  be  replaced:  2 
Reported  annual  cost  of  maintaining  legacy  systems:  $25  million 
Number  of  system  interfaces:  27 

Date  of  last  certification  of  funding:  September  2,  2009,  by  the  DBSMC 
Number  of  system  users:  21 ,000 
Number  of  locations:  104 

Source:  DOD’s  LMP  Program  Management  Office.  These  data  have  not  been  validated. 


$2,630  billion 
$  637  million 
$1 .993  billion 
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Program  Status 


LMP  was  deployed  at  the  Army  Communications-Electronics  Command 
and  Tobyhanna  Army  Depot  in  July  2003.  In  May  2009,  the  second 
deployment  of  LMP  became  operational  at  the  Army  Aviation  and  Missile 
Command  and  Corpus  Christi  and  Letterkenny  Army  Depots.  The  final 
deployment  of  LMP  is  scheduled  to  occur  in  October  2010  at  the  Army 
Sustainment  Command,  the  Joint  Munitions  and  Lethality  Command,  the 
Tank-automotive  and  Armaments  Command,  and  the  Anniston  and  Red 
River  Army  Depots. 

LMP  has  experienced  schedule  slippages  primarily  because  requirements 
management30  and  system  testing  were  ineffective  which  we  reported  on  in 
May  200431  and  June  2005.32  For  example,  at  the  Tobyhanna  Army  Depot 
deployment  in  fiscal  year  2003,  customers  were  not  being  properly  billed 
for  work  performed  which  affected  the  accurate  recording  of  revenue,  and 
account  balances  could  not  be  reconciled  when  transferred  from  the 
legacy  systems  to  LMP.  As  a  result,  the  full  deployment  date  of  the  system 
has  slipped  by  6  years. 

Furthermore,  in  April  20 10, 33  we  reported  that  the  Army’s  management 
processes  for  ensuring  data  reliability  that  were  established  prior  to  the 
second  deployment  of  LMP  were  not  effective.  Specifically,  the  Army  was 
unable  to  ensure  that  the  data  used  by  LMP  were  of  sufficient  quality  to 
enable  the  depots  to  perform  their  day-to-day  missions  after  LMP  became 
operational.  As  a  result  of  these  data  quality  issues,  depot  personnel  had  to 
develop  and  use  manual  work-around  processes  until  they  could  correct 
the  data  in  LMP,  which  prevented  the  Army  from  achieving  the  expected 
benefits  from  LMP.  Data  quality  issues  occurred  despite  improvements 
made  by  the  Army  to  address  similar  issues  experienced  during  the  first 
deployment  of  LMP  because  the  Army’s  testing  strategy  did  not  provide 


30 According  to  the  Software  Engineering  Institute,  requirements  management  is  a  process 
that  establishes  a  common  understanding  between  the  customer  and  the  software  project 
manager  regarding  the  customer’s  business  needs  that  will  be  addressed  by  a  project.  A 
critical  part  of  this  process  is  to  ensure  that  the  requirement  development  portion  of  the 
effort  documents,  at  a  sufficient  level  of  detail,  the  problems  that  need  to  be  solved  and  the 
objectives  that  need  to  be  achieved. 

31GAO,  DOD  Business  Systems  Modernization:  Billions  Continued  to  be  Invested  with 
Inadequate  Management  Oversight  and  Accountability,  GAO-04-615  (Washington,  D.C.: 
May  27,  2004). 

32GAO,  Aumy  Depot  Maintenance:  Ineffective  Oversight  of  Depot  Maintenance  Operations 
and  System  Implementation  Efforts,  GAO-05-441  (Washington,  D.C.:  June  30,  2005). 

33GAO-10-461. 
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reasonable  assurance  that  the  data  being  used  by  LMP  were  accurate  and 
reliable.  We  made  recommendations  to  help  improve  the  third  deployment 
of  LMP.  We  are  following  up  on  the  Army’s  efforts  to  implement  our 
recommendations  and  will  report  on  those  actions  separately. 

The  PMO  further  noted  that  the  original  life-cycle  cost  estimate  of 
approximately  $2.6  billion34  covering  fiscal  years  2000  through  2021 
remained  unchanged  as  of  December  2009.  PMO  officials  told  us  that  there 
were  no  issues  or  conditions  that  had  been  placed  upon  LMP  by  the  MDA, 
IRBs  or  the  DBSMC  that  needed  to  be  resolved  as  of  December  2009. 


34At  the  time  LMP  was  designated  as  a  MAIS  program  in  December  2007,  it  was  required  to 
comply  with  the  DOD  guidance  for  MAIS  programs.  This  guidance  requires,  among  other 
things,  that  a  MAIS  program  have  a  completed  and  approved  acquisition  program 
baseline — the  baseline  description  of  the  program,  including  the  life-cycle  cost  estimate — 
prior  to  Milestone  B  approval.  The  $2.6  billion  is  the  only  life-cycle  cost  estimate  that  has 
been  developed  for  the  program. 
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Navy  Enterprise  Resource 
Planning  System 


Program  Status 


DOD  Program  Data  Provided  for  Navy  ERP,  as  of  December  31, 2009 

Date  of  Initiation:  July  2003 

Program  owner:  Assistant  Secretary  of  the  Navy,  Research,  Development,  and 
Acquisition 

Reported  life-cycle  cost  estimate:  $2.4  billion 

•  Development  and  Modernization  $1.0  billion 

•  Operations  and  Maintenance  $1.4  billion 

Reported  amount  expended:  $691.3  million 

Reported  legacy  systems  to  be  replaced:  98 

Reported  annual  cost  of  maintaining  legacy  systems:  $102  million 

Number  of  system  interfaces:  51 

Date  of  last  certification  of  funding:  September  2,  2009,  by  the  DBSMC 
Number  of  system  users:  66,000 
Number  of  locations:  53 

Source:  DOD’s  Navy  ERP  Program  Management  Office.  These  data  have  not  been  validated. 


Navy  ERP  is  to  be  implemented  in  two  phases.  As  part  of  phase  1,  the 
financial  and  acquisition  functionalities  of  Navy  ERP  were  deployed  to  the 
Naval  Air  Systems  Command,  Naval  Supply  Systems  Command,  and  the 
Space  and  Naval  Warfare  Systems  Command.  Those  functionalities  are 
scheduled  for  deployment  for  the  general  fund  at  the  Naval  Sea  Systems 
Command  in  October  2010  and  the  Navy  Working  Capital  Fund  in  October 
2011  and  the  Office  of  Naval  Research  and  Strategic  Systems  Planning  in 
October  2012.  Phase  2  is  currently  in  progress  with  the  deployment  of  the 
wholesale  and  retail  supply  functionalities  to  the  Navy.  According  to  the 
PMO,  Navy  ERP  is  currently  being  used  by  38,000  users  and  is  executing 
approximately  $37  billion  of  the  Navy’s  total  obligational  authority. 
Further,  the  PMO  noted  that  in  fiscal  year  2010,  19  legacy  systems  have 
already  been  retired. 

The  Navy  ERP  implementation  has  experienced  slippages  of  2  years. 
Originally,  the  Navy  ERP  was  to  achieve  full  deployment  in  fiscal  year 
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2011,  but  now  full  deployment  is  planned  for  fiscal  year  2013.  According  to 
program  documentation,  these  slippages  occurred,  in  part,  because  of 
problems  experienced  in  data  conversion  and  adopting  new  business 
procedures  associated  with  implementing  the  ERP.  The  delay  occurred  at 
the  Naval  Air  Systems  Command  and  affected  the  deployment  schedule  for 
the  other  locations.  In  addition  to  slippages  in  schedule,  there  have  also 
been  increases  in  the  life-cycle  cost  estimate.  The  2003  original  life-cycle 
cost  estimate  for  the  Navy  ERP  was  about  $1.87  billion.  This  estimate  was 
later  revised  in  August  2004,  December  2006,  and  again  in  September  2007 
to  $2.4  billion.  According  to  the  September  2007  acquisition  program 
baseline,  the  estimated  $2.4  billion  is  for  acquisition,  operations,  and 
support  for  fiscal  years  2004  through  2023.  Moreover,  in  September  2008, 35 
we  reported  that  not  effectively  implementing  key  IT  management 
controls,  such  as  earned  value  management,  has  contributed  to  the  more 
than  2-year  schedule  delay  and  almost  $600  million  cost  overrun  on  the 
program  since  it  began,  and  will  likely  contribute  to  future  delays  and 
overruns  if  not  corrected. 

The  IRB  has  identified  two  issues  or  conditions  that  the  Navy  ERP  PMO 
has  to  address:  (1)  provide  a  description  of  how  the  Navy  plans  to  use  the 
item-unique  identification  (IUID)36  and  (2)  provide  an  updated  checklist  to 
BTA  showing  compliance  with  the  Standard  Financial  Information 
Structure  (SFIS).37  The  PMO  stated  that  it  presented  a  plan  to  the  Navy 
Comptroller  in  February  2010  describing  how  it  will  use  IUID  and  provided 
BTA  the  SFIS  checklist  in  April  2010. 


3BGAO-08-896. 

36 According  to  DOD,  the  purpose  of  the  IUID  is  to  facilitate  asset  accountability  and 
tracking,  including  the  identification  and  aggregation  of  related  costs  to  derive  the  full  cost 
of  a  contract  deliverable. 

3iBTA  defines  SFIS  as  a  comprehensive  “common  business  language”  that  supports 
information  and  data  requirements  for  budgeting,  financial  accounting,  cost/performance 
management,  and  external  reporting  across  the  DOD  enterprise. 
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Global  Combat  Support 
System-Marine  Corps 


Program  Status 


DOD  Program  Data  for  GCSS-MC,  as  of  December  31 , 2009 

Date  of  Initiation:  September  2003 

Program  owner:  Assistant  Secretary  of  the  Navy,  Research,  Development,  and 

Acquisition 

Reported  life-cycle  cost  estimate: 

$934  million 

•  Development  and  Modernization 

$489  million 

•  Operations  and  Maintenance 

$445  million 

Reported  amount  expended:  $245  million 

Reported  legacy  systems  to  be  replaced:  4 

Reported  annual  cost  of  maintaining  legacy  systems:  $4.5  million 

Number  of  system  interfaces:  42 

Date  of  last  certification  of  funding:  June  1 , 201 0  by  the  DBSMC 

Number  of  system  users:  33,000 

Number  of  locations:  6 

Source:  DOD’s  GCSS-MC  Program  Management  Office.  These  data  have  not  been  validated. 


GCSS-MC  was  authorized  to  “Go  Live”  for  field  user  evaluation  in  March 
2010  and  Milestone  C  was  granted  in  May  2010.  GCSS-MC  is  to  be 
implemented  in  three  phases.  Phase  1  is  intended  to  provide  a  wide  range 
of  asset  management  capabilities  such  as  planning  inventory  requirements 
to  support  current  and  future  demands;  requesting  and  tracking  the  status 
of  products  (e.g.,  supplies  and  personnel)  and  services  (e.g.,  maintenance 
and  engineering);  allocating  resources  (e.g.,  inventory,  warehouse 
capacity,  and  personnel)  to  support  unit  demands  for  specific  products; 
and  scheduling  maintenance  resources  (e.g.,  manpower,  equipment,  and 
supplies)  for  specific  assets,  such  as  vehicles.  Phases  2  and  3  are  intended 
to  provide  additional  functionally  such  as  transportation  and  wholesale 
inventory  management. 

To  date,  there  have  been  program  slippages  and  cost  increases.  The  PMO 
told  us  that  full  deployment  for  phase  1  was  originally  scheduled  to  be 
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achieved  in  November  2009.  However,  the  current  estimated  full 
deployment  date  for  phase  1  is  January  2013.38  GCSS-MC  program  officials 
informed  us  that  the  schedule  slippage  for  phase  1  occurred  incrementally 
over  time  during  the  design,  build,  and  test  phases  of  the  program.  The 
slippages  occurred  because  of  issues  associated  with  system  interfaces 
and  the  conversion  of  data  from  the  legacy  systems.  Moreover,  in  July 
2008, 39  we  reported  that  not  effectively  implementing  key  IT  management 
controls,  such  as  economically  justifying  investment  in  the  system,  has  in 
part  contributed  to  a  3-year  schedule  slippage  and  about  $193  million  cost 
overrun  on  the  first  phase  of  the  program  and  will  likely  contribute  to 
future  delays  and  overruns  if  not  corrected. 

These  schedule  slippages  caused  the  program  to  exceed  the  MAIS  critical- 
breach  criteria  for  l  ime-certain  development,  which  is  the  failure  to 
achieve  initial  operating  capability  within  5  years  of  Milestone  A  approval. 
PMO  officials  also  told  us  that  initially,  GCSS-MC  had  an  estimated  cost  of 
approximately  $126  million  over  a  7-year  life  cycle.40  This  cost  estimate 
was  later  revised  in  2005  to  approximately  $249  million  over  a  13-year  life 
cycle.41  Currently,  the  PMO  estimates  the  total  life-cycle  cost  estimate  for 
phase  1  to  be  approximately  $934  million.  The  total  life-cycle  cost  estimate 
for  the  additional  phases  has  not  been  determined.  According  to  the  PMO, 
phase  2  is  in  the  preliminary  planning  stage  and  all  additional  phases  will 
have  separate  acquisition  program  baselines.42  As  a  result,  a  total  life-cycle 
cost  estimate  for  the  entire  system  may  not  be  available  for  several  years. 

The  IRB  directed  that  the  GCSS-MC  PMO  (1)  provide  a  component-wide 
plan  that  addresses  how  GCSS-MC  will  include  the  capability  to  use  IUID 


38January  2013  is  the  estimated  full  deployment  date  in  the  proposed  acquisition  program 
baseline  submitted  for  MDA  approval  in  February  2010. 

39GAO-08-822. 

40 According  to  the  May  10,  2004,  analysis  of  alternatives,  this  estimate  was  a  “rough  order 
of  magnitude”  for  research  and  development,  procurement  and  operations  and  support 
from  fiscal  years  2004  through  2011. 

41According  to  the  July  15,  2005,  economic  analysis,  program  costs  are  estimated  from 
fiscal  years  2005  through  2018,  in  base  year  2005  dollars,  and  exclude  $9.6  million 
associated  with  supporting  and  maintaining  legacy  systems  during  GCSS-MC  development 
and  $11.9  million  in  fiscal  year  2004  sunk  costs. 

42The  acquisition  program  baseline  is  an  important  document  for  program  management  and 
should  reflect  the  approved  program  being  executed.  In  this  regard,  the  acquisition 
program  baseline  formally  documents  the  program’s  estimated  cost,  schedule,  and 
performance  goals. 
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and  (2)  brief  the  Navy  DCMO  on  the  extent  to  which  business  process 
reengineering  (BPR)  has  been  performed  to  address  the  statutory 
requirement  regarding  BPR  in  Section  1072  of  the  Fiscal  Year  2010 
National  Defense  Authorization  Act.  In  this  regard,  the  act  directs  the 
Chief  Management  Officer  to  determine  whether  or  not  appropriate 
business  process  re-engineering  efforts  have  been  undertaken  to  ensure 
that  (1)  the  business  process  to  be  supported  by  the  business  system  will 
be  as  streamlined  and  efficient  as  practicable  and  (2)  the  need  to  tailor  the 
ERP  to  meet  unique  requirements  or  incorporate  unique  interfaces  has 
been  eliminated.  The  PMO  stated  that  it  provided  the  BPR  information  to 
the  Navy  DCMO  and  the  DCMO  indicated  that  the  PMO  had  addressed  the 
requirements  contained  in  the  act. 


Defense  Enterprise 
Accounting  and 
Management  System 


DOD  Program  Data  for  DEAMS,  as  of  December  31 , 2009 

Date  of  initiation:  August  2003 

Program  owner:  Assistant  Secretary  of  the  Air  Force  for  Financial  Management  and 

Comptroller 

Reported  life-cycle  cost  estimate: 

$2,048  billion 

•  Development  and  Modernization 

$1 .030  billion 

•  Operations  and  Maintenance 

$1,018  billion 

Reported  amount  expended:  $139.1  million 

Reported  legacy  systems  to  be  replaced:  10 

Reported  annual  cost  of  maintaining  legacy  systems:  $55.9  million 

Number  of  system  interfaces:  100 

Date  of  last  certification  of  funding:  December  14,  2009,  by  the  DBSMC 

Number  of  system  users:  30,000 

Number  of  locations:  179 

Source:  DOD’s  DEAMS  Program  Management  Office.  These  data  have  not  been  validated. 


Program  Status  DEAMS  will  be  deployed  in  three  phases.  Phase  1  deployed  limited 

functionality — recording  commitments — to  about  650  system  users  at 
Scott  Air  Force  Base  in  July  2007.  According  to  the  PMO,  as  part  of  phase 
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1,  additional  functionality  was  deployed  to  an  additional  870  users  in  May 
2010.  Further,  the  PMO  noted  that  DEAMS  is  currently  scheduled  to 
achieve  initial  operating  capability  for  phase  2  for  the  U.S.  Transportation 
Command  and  most  of  the  Air  Force’s  major  commands  in  fiscal  year 
2014.  According  to  the  PMO,  the  final  phase  of  DEAMS  will  be  deployed  to 
the  remaining  Air  Force’s  major  commands  by  fiscal  year  2017,  thereby 
providing  the  entire  spectrum  of  general  fund  capabilities  to  the  entire  Air 
Force. 

The  Air  Force  expects  DEAMS  to  reach  full  deployment  in  fiscal  year 
2017 — which  is  a  3-year  slippage  from  the  full  deployment  date  reported  at 
program  initiation.  According  to  the  PMO,  DEAMS  has  experienced  a 
3-year  schedule  slippage  because  of  problems  caused  by  software  code 
defects,  integration  test  delays  and  to  accommodate  schedule  risk. 

DEAMS  program  management  officials  acknowledged  that  the 
standardization  of  computer  desktops  across  the  Air  Force  contributed  to 
schedule  slippages.  Our  August  2008  report  discussed  this  specific 
problem.43 

In  addition  to  schedule  slippages,  DEAMS  also  had  an  increase  in  its 
life-cycle  cost  estimate.  In  August  2008,  we  reported  that  the  Air  Force’s 
life-cycle  cost  estimate  for  DEAMS  was  about  $1.1  billion  through  fiscal 
year  202 1.44  According  to  the  PMO,  as  of  December  2009,  the  life-cycle  cost 
estimate  for  the  DEAMS  is  approximately  $2  billion  through  fiscal  year 
2027.  The  PMO  stated  that  the  increase  in  the  life-cycle  cost  estimate  can 
be  attributed  to  changes  in  the  program  implementation  strategy  from  two 
phases  to  three  phases  and  program  development  and  testing  issues. 

The  IRB  directed  the  DEAMS’s  PMO  to  (1)  create  an  IUID  compliance  plan 
indicating  when  the  system  will  include  the  capability  to  use  IUID, 

(2)  identify  the  date  that  one  of  the  legacy  systems  will  be  subsumed, 

(3)  provide  a  plan  on  how  DEAMS  will  meet  Environmental  Liabilities 
Recognition  Valuation  and  Reporting  requirements,  and  (4)  comply  with 
Section  1072  of  the  National  Defense  Authorization  Act  for  Fiscal  Year 
2010  related  to  business  process  reengineering.  According  to  the  PMO,  the 
Air  Force  has  addressed  these  issues. 


*^0-08-866. 

^GAO-os-see. 


Page  28 


GAO-11-53  DOD  Business  Systems 


Expeditionary  Combat 
Support  System 


Program  Status 


DOD  Program  Data  for  ECSS,  as  of  December  31 , 2009 

Date  of  Initiation:  January  2004 

Program  owner:  Deputy  Chief  of  Staff  for  Logistics,  Installations,  and  Mission 
Support,  Headquarters,  U.S.  Air  Force 

Reported  life-cycle  cost  estimate:  $5.2  billion 

•  Development  and  Modernization  $3.4  billion 

•  Operations  and  Maintenance  $1 .8  billion 

Reported  amount  expended:  $518.9  million 

Reported  legacy  systems  to  be  replaced:  240 

Reported  annual  cost  of  maintaining  legacy  systems:  $325  million 

Number  of  system  interfaces:  157  (phase  1)  and  673  (phases  2,  3,  and  4) 

Date  of  last  certification  of  funding:  September  2,  2009,  by  the  DBSMC 

Number  of  system  users:  250,000 

Number  of  locations:  1 86 

Source:  DOD’s  ECSS  Program  Management  Office.  These  data  have  not  been  validated. 


ECSS  will  be  deployed  in  four  phases.  The  Air  Force  anticipates  that  phase 
1  will  begin  deployment  in  June  2012,  with  phase  2  scheduled  for 
deployment  in  April  2014,  phase  3  in  January  2015,  and  phase  4  in 
November  2015.  According  to  the  PMO,  each  phase  will  provide  additional 
functionality  to  the  system  users.  Phase  1  will  focus  on  base  materiel  and 
equipment  management,  phase  2  will  concentrate  on  global  materiel  and 
equipment  management  and  enterprise  planning,  phase  3  will  involve 
depot  maintenance  repair  and  overhaul,  and  phase  4  will  involve  flight  line 
maintenance  and  ammunition  management.  The  PMO  estimated  that  full 
deployment  will  be  achieved  in  July  2016 — a  slippage  of  at  least  4  years. 
According  to  the  PMO,  the  slippage  can  be  attributed  to  (1)  two  contract 
award  protests — both  denied  by  GAO — and  (2)  the  change  in  the 
implementation  strategy,  which  had  originally  called  for  the  system  to  be 
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implemented  in  three  phases.  Also,  in  our  August  2008  report,45  we  noted 
that  the  life-cycle  cost  estimate  was  approximately  $3  billion  for  the  entire 
ECSS  program  when  it  was  scheduled  for  three  phases.  According  to  the 
ECSS  PMO,  the  current  life-cycle  cost  estimate  is  approximately 
$5.2  billion.  Funding  has  not  yet  been  approved  for  phases  2  through  4. 

The  PMO  noted  that  ECSS  will  seek  approval  at  each  phase’s  critical 
milestone  in  order  to  go  forward  to  the  next  phase. 

The  Air  Force  DCMO  told  us  that  Air  Force  leadership  (including  the 
Secretary  of  the  Air  Force,  Air  Force  Chief  of  Staff,  and  Senior  Acquisition 
Executive)  reviewed  the  program  to  determine  whether  it  should  be 
restructured  or  cancelled.  The  leadership  was  specifically  concerned 
about  the  size,  scope,  and  pace  of  the  program.  The  program  was 
restructured,  and  in  June  2009,  the  decision  was  made  to  pursue  only  the 
revised  phase  1  pending  a  demonstration  of  the  program’s  ability  to  deliver 
to  the  revised  schedule.  The  DCMO  told  us  that  the  Air  Force  will  make  a 
decision  on  (1)  whether  to  implement  phase  1  and  (2)  whether  to  budget 
for  the  other  phases  in  June  2010.  According  to  the  PMO,  it  anticipates  the 
Air  Force  fully  funding  phase  1  and  the  long-lead  requirements  for  phase  2 
in  the  fiscal  year  2012  program  objective  memorandum.46 

Because  of  changes  in  the  implementation  strategy,  in  September  2009,  the 
DOD  MDA  approved  a  revised  Milestone  A  for  ECSS.  The  revised 
milestone  provides  for  additional  funding,  and  it  grants  the  Air  Force 
authority  to  continue  with  ECSS  technology  development  and  prepare  for 
Milestone  B  for  phase  1.  In  preparing  for  Milestone  B,  the  Air  Force  was 
directed  to 

•  present  quarterly  reports  regarding  the  progress  of  the  program,  including 
internal  and  external  challenges  and  risks,  to  the  IRBs  for  weapons 
system,  material,  service,  and  financial  management; 

•  complete  an  enterprise  risk  assessment  methodology  review  of  the 
program  120  days  prior  to  Milestone  B;  and 

•  provide  a  cost  analysis  requirement  document  to  the  Air  Force  Analysis 
Agency  to  support  the  development  of  an  independent  cost  estimate. 


45GAO-08-866. 

46The  program  objective  memorandum  details  planned  resource  allocation  6  years  in  the 
future. 


Page  30 


GAO-11-53  DOD  Business  Systems 


According  to  the  PMO,  each  of  these  actions  was  completed  by  May  20, 
2010. 
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Service  Specific  Integrated 
Personnel  and  Pay 
Systems47 


DOD  Program  Data  for  Service  Specific  Integrated  Personnel 
and  Pay  Systems,  as  of  December  31 , 2009 

Date  of  Initiation:  February  1998 

Program  owner:  Army — Army’s  Program  Executive  Office,  Enterprise  Information  Systems 

Navy — Chief  of  Naval  Operations 

Air  Force — Air  Force  Program  Executive  Office  and  Service  Acquisition  Executive 

Reported  life-cycle  cost  estimate:  Army— Has  not  yet  been  determined 

Navy — $1 .3  billion 
Air  Force — $1 .7  billion 

Reported  amount  expended:  $841 .1  million 

Legacy  systems  to  be  replaced:  Army— 65 

Navy — 7 

Air  Force — Has  not  yet  been  determined 

Reported  annual  cost  of  maintaining  legacy  systems:  Army — $39  million 

Navy — $69  million 

Air  Force —  Has  not  yet  been  determined 

Number  of  system  interfaces:  Has  not  yet  been  determined 

Date  of  last  certification  of  funding:  Not  applicable  for  the  military  services  as  of  December  2009 
Number  of  system  users:  Has  not  yet  been  determined 
Number  of  locations:  Has  not  yet  been  determined 

Sources:  The  Army,  Navy,  and  Air  Force  program  management  offices.  These  data  have  not  been  validated. 


4 'Each  military  department  refers  to  its  respective  personnel  and  pay  system  by  a  different 
name — the  Integrated  Personnel  and  Pay  System — Army,  the  Navy  Future  Pay  and 
Personnel  Solution,  and  the  Air  Force  Integrated  Personnel  and  Pay  System.  For  purposes 
of  this  report,  we  are  collectively  referring  to  these  efforts  as  the  Service  Specific 
Integrated  Personnel  and  Pay  Systems — a  name  used  by  DOD. 
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Program  Status 


Integrated  Personnel  and  Pay 
System-Army  (IPPS-A) 


Navy  Future  Pay  and  Personnel 
Solution 


In  a  January  2009  memorandum,  the  Deputy  Secretary  of  Defense  changed 
the  department’s  strategy  for  implementing  an  integrated  personnel  and 
pay  system.  The  memorandum  directed  the  BTA  to  develop  the  pay 
module  and  provide  it  to  the  military  departments.  Each  military 
department  would  be  responsible  for  implementing  an  integrated 
personnel  and  pay  system  for  its  respective  service.  In  revising  the 
department’s  strategy,  a  subsequent  memorandum  issued  September  2009 
by  the  Under  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics) 
noted  that  the  capabilities  needed  by  DOD  to  develop  integrated  personnel 
and  pay  systems  are  best  met  through  the  military  departments  because  of 
several  risks,  including  governance,  technical  complexities,  and  past  failed 
attempts  of  developing  DIMHRS  as  a  one-fits-all  solution.  The 
memorandum  further  noted  that  military  departments  were  to  use,  to  the 
maximum  extent  practical,  the  DIMHRS  requirements  related  to  the  pay 
module  developed  by  BTA.  Highlighted  below  is  the  status  of  each  of  the 
military  department’s  efforts  to  implement  an  integrated  personnel  and 
pay  system. 

Army  PMO  officials  told  us  that  in  accordance  with  the  September  2009 
memorandum,  the  Army  intends  to  use  the  BTA-developed  pay  module, 
develop  the  personnel  module  and  implement  an  integrated  system.  Once 
IPPS-A  is  developed  it  will  be  implemented  in  several  phases.  The  first 
deployment  is  planned  for  the  Army  National  Guard,  followed  by  the  Army 
Reserves,  and  then  the  active  Army.  The  PMO  stated  that  the  personnel 
and  pay  portion  will  be  deployed  to  all  Army  components  by  August  2014. 
The  Army  anticipates  that  full  deployment  will  occur  late  in  fiscal  year 
2014.  The  PMO  informed  us  that  the  Army  is  in  the  process  of  developing 
the  life-cycle  cost  estimate. 

According  to  PMO  officials,  the  Navy  is  in  the  process  of  evaluating  the 
extent  to  which  the  BTA-developed  pay  module  can  be  used  to  meet  its 
needs  for  an  integrated  system.  Navy  anticipates  that  this  evaluation  will 
be  completed  by  the  second  quarter  of  fiscal  year  2011.  PMO  officials  told 
us  that  if  the  pay  module  can  be  used,  the  system  will  be  implemented  in 
two  phases.  Phase  1  will  consolidate  the  existing  legacy  personnel  systems 
and  establish  a  single  personnel  record.  Phase  2  will  be  the 
implementation  of  the  pay  module.  Navy  would  begin  deployment  in  fiscal 
year  2014  for  phase  1  and  fiscal  year  2015  for  phase  2,  with  full  deployment 
being  achieved  in  fiscal  year  2017.  PMO  officials  told  us  that  the  Navy 
estimates  that  the  life-cycle  cost  estimate  for  its  integrated  personnel  and 
pay  system  will  be  about  $1.3  billion.  The  PMO  further  stated  that  if  an 
alternative  to  using  the  BTA-developed  pay  module  is  selected,  the 
implementation  dates  and  estimated  cost  may  change.  In  the  September 
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Air  Force  Integrated  Personnel 
and  Pay  System 


2009  memorandum,  it  was  noted  that  the  Marine  Corps  will  continue  to 
use  the  Marine  Corps  Total  Force  System  because  it  is  already  an 
integrated  personnel  and  pay  system. 

At  the  time  of  our  review,  Air  Force  was  evaluating  the  BTA-developed  pay 
module  to  assess  whether  it  could  be  used.  According  to  the  PMO,  the 
system  will  be  implemented  in  three  phases,  provided  the  existing  BTA- 
developed  pay  module  can  be  used.  Phase  1  will  consist  of  transferring 
data  from  the  legacy  systems  to  the  new  integrated  personnel  and  pay 
system  and  will  include  implementation  of  leave/benefits  for  all.  Phase  2 
will  provide  an  integrated  personnel  and  pay  solution  for  active  Air  Force 
officers,  and  phase  3  will  deploy  the  system  to  the  rest  of  the  Air  Force 
personnel  including  guard  and  reserve  personnel.  The  quantity  and 
content  of  the  phases  may  change  as  the  Air  Force  evolves  the  acquisition 
and  deployment  strategies.  According  to  the  PMO,  it  is  anticipated  that  full 
deployment  will  be  achieved  in  April  2018.  The  Air  Force  PMO  currently 
estimates  the  life-cycle  cost  estimate  to  be  about  $1.7  billion  covering 
fiscal  year  2010  through  fiscal  year  2027.  The  PMO  told  us  that  as  the  Air 
Force  better  defines  its  implementation  strategy,  the  implementation  dates 
and  life-cycle  cost  estimate  could  change.  The  PMO  also  said  that  the  Air 
Force  is  in  the  process  of  ascertaining  how  many  legacy  systems  can  be 
eliminated  through  its  implementation  of  an  integrated  personnel  and  pay 
system. 
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Defense  Agencies  Initiative 


DOD  Program  Data  for  DAI,  as  of  December  31 , 2009 

Date  of  Initiation:  January  2007 

Program  owner:  The  Business  Transformation  Agency  was  the  first  entity  to 
implement  DAI.  Each  defense  agency  will  be  responsible  for  the  management  and 
oversight  of  its  respective  implementation. 

Reported  life-cycle  cost  estimate:  Has  not  yet  been  determined 

•  Development  and  Modernization  Has  not  yet  been  determined 

•  Operations  and  Maintenance  Has  not  yet  been  determined 

Reported  amount  expended:  $40.2  million 

Reported  legacy  systems  to  be  replaced:  17 

Reported  annual  cost  of  maintaining  legacy  systems:  $35  million 

Number  of  system  interfaces:  24 

Date  of  last  certification  of  funding:  September  30,  2009  by  the  DBSMC 
Number  of  system  users:  15,000  (estimated) 

Number  of  locations:  1 1  (estimated) 

Source:  DOD’s  DAI  Program  Management  Office.  These  data  have  not  been  validated. 


Program  Status  DAI  became  operational  at  BTA  in  October  2008  and  at  the  Defense 

Technical  Information  Center  in  October  2009.  Table  3  lists  the  defense 
agencies  that  are  scheduled  to  implement  DAI  in  fiscal  years  2011  through 
2013. 
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Table  3:  Defense  Agencies’  Scheduled  Implementation  of  DAI 

Defense  agency 

Fiscal  year 
2011 

Fiscal  year 
2012 

Fiscal  year 
2013 

Uniform  Services  University  of  the  Health 
Services 

X 

Missile  Defense  Agency 

X 

Defense  Threat  Reduction  Agency 

X 

Defense  Information  Systems  Agency 

X 

Defense  Technology  Security  Administration 

X 

Chemical  Biological  Defense  Program 

X 

TRICARE  Management  Agency — 
Headquarters 

X 

Defense  Media  Agency 

X 

Defense  Information  System  Agency — 
General  Fund 

X 

Defense  Acquisition  University 

X 

Defense  POW/Missing  Personnel  Office 

X 

Defense  Advanced  Research  Projects 

Agency 

X 

Defense  Security  Service 

X 

Office  of  Economic  Adjustment 

X 

Center  for  Countermeasures 

X 

National  Defense  University 

X 

Source:  Business  Transformation  Agency. 


There  has  been  some  slippage  in  the  implementation  schedule.  However, 
at  the  time  of  our  review,  a  revised  full  deployment  date  for  all  of  the 
agencies  scheduled  to  use  DAI  had  not  been  established.  According  to  the 
department’s  fiscal  year  2011  IT  budget  request,  additional  defense 
agencies  have  expressed  an  interest  in  using  DAI.  However,  the  Financial 
Management  IRB  and  the  DBSMC  must  grant  approval  to  any  entity  that 
wants  to  use  DAI.  DOD’s  budget  request  notes  that  the  total  cost  of  the 
program  is  affected  by  the  number  of  agencies  participating.  The  budget 
request  further  notes  that  a  more  accurate  implementation-plus- 
sustainment  cost  can  be  determined  once  all  of  the  signed  memorandums 
of  intent  from  agencies  wanting  to  use  DAI  have  been  received. 
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DOD  Did  Not  Follow 
Key  Best  Practices  for 
Estimating  ERP 
Schedules  and  Cost, 
Resulting  in 
Unreliable  Estimates 


Our  analysis  of  the  schedules  and  cost  estimates  for  four  ERP  programs — 
DEAMS,  ECSS,  GFEBS,  and  GCSS-Army — found  that  none  of  the 
programs  are  fully  following  best  practices  for  developing  reliable 
schedules  and  cost  estimates.  More  specifically,  none  of  the  programs  had 
developed  a  fully  integrated  master  schedule  (IMS)  that  reflects  all 
activities,  including  both  government  and  contractor  activities.  In  addition, 
none  of  the  programs  established  a  valid  critical  path  or  conducted  a 
schedule  risk  analysis.48  We  have  previously  reported  that  the  schedules 
for  GCSS-MC  and  Navy  ERP  were  developed  using  some  of  these  best 
practices,  but  several  key  practices  were  not  fully  employed  that  are 
fundamental  to  having  a  schedule  that  provides  a  sufficiently  reliable  basis 
for  estimating  costs,  measuring  progress,  and  forecasting  slippages.49  We 
recommended  that  each  program  follow  best  practices  to  update  its 
respective  schedule.  DOD  generally  agreed  with  the  recommendations. 
Additional  details  on  the  status  of  the  recommendations  are  discussed  in 
appendix  III.  The  success  of  any  program  depends  on  having  a  reliable 
schedule  of  the  program’s  work  activities  that  will  occur,  how  long  they 
will  take,  and  how  the  activities  are  related  to  one  another.  As  such,  the 
schedule  not  only  provides  a  road  map  for  systematic  execution  of  a 
program,  but  also  provides  the  means  by  which  to  gauge  progress,  identify 
and  address  potential  problems,  and  promote  accountability. 


Our  analysis  of  the  four  programs’  cost  estimates  found  that  ECSS, 
GFEBS,  and  GCSS-Army  did  not  include  a  sensitivity  analysis,  while  cost 
estimates  for  GFEBS  did  not  include  a  risk  and  uncertainty  analysis.  GAO, 
OMB,  and  DOD  guidance50  stipulate  that  risk  and  uncertainty  analysis 
should  be  performed  to  determine  the  level  of  risk  associated  with  the 
dollar  estimate.  Furthermore,  a  sensitivity  analysis  would  assist  decision 
makers  in  determining  how  changes  to  assumptions  or  key  cost  drivers 
(such  as  labor  or  equipment)  could  affect  the  cost  estimate.  We  have 
previously  reported  that  the  cost  estimates  for  Navy  ERP  and  GCSS-MC 
are  comprehensive  and  well-documented,  but  only  partially  accurate  and 


48A  critical  path  is  the  longest  duration  path  through  a  sequenced  list  of  activities  within  a 
schedule.  A  schedule  risk  analysis  uses  statistical  techniques  to  predict  a  level  of 
confidence  in  meeting  a  completion  date. 

49GAO-08-822  and  GAO-08-896. 

50GAO-09-3SP;  OMB  Revised  Circular  No.  A-94,  Guidelines  and  Discount  Rates  for  Benefit- 
Cost.  Analysis  of  Federal  Programs  (Oct.  29,  1992);  and  DOD  Instruction  7041.3,  Economic 
Analysis  of  Decisionmaking  (Nov.  7,  1995). 
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credible.51  We  recommended  that  each  program  update  its  respective  cost 
estimate  following  best  practices.  The  department  generally  agreed  with 
the  recommendations.  Additional  details  on  the  status  of  the 
recommendations  are  discussed  in  appendix  III.  For  DOD  management  to 
make  good  decisions,  the  program  estimate  must  reflect  the  degree  of 
uncertainty  so  that  a  level  of  confidence  can  be  given  about  the  estimate.  A 
reliable  cost  estimate  provides  the  basis  for  informed  investment  decision 
making,  realistic  budget  formulation  and  program  resourcing,  meaningful 
progress  measurement,  proactive  course  correction,  and  accountability  for 
results. 


Program  Schedules  Not 
Developed  in  Accordance 
with  Key  Scheduling 
Practices 


Our  cost  guide  best  practices  and  related  federal  guidance  call  for  a 
program  schedule  to  be  programwide,  meaning  that  it  should  include  an 
integrated  breakdown  of  the  work  to  be  performed  by  both  the 
government  and  its  contractors  over  the  expected  life  of  the  program.52 
Our  guidance  identifies  nine  scheduling  best  practices  that  are  integral  to  a 
reliable  and  effective  master  schedule:  (1)  capturing  all  activities, 

(2)  sequencing  all  activities,  (3)  assigning  resources  to  all  activities, 

(4)  establishing  the  duration  of  all  activities,  (5)  integrating  schedule 
activities  horizontally  and  vertically,  (6)  establishing  the  critical  path  for 
all  activities,  (7)  identifying  float  between  activities,  (8)  conducting  a 
schedule  risk  analysis,  and  (9)  updating  the  schedule  using  logic  and 
durations  to  determine  the  dates. 


The  scheduling  best  practices  are  interrelated  so  that  deficiencies  in  one 
best  practice  will  cause  deficiencies  in  other  best  practices.  For  example, 
if  the  schedule  does  not  capture  all  activities,  then  there  will  be 
uncertainty  about  whether  activities  are  sequenced  in  the  correct  order 
and  whether  the  schedule  properly  reflects  the  resources  needed  to 
accomplish  the  work.  The  schedule  should  use  logic  and  durations  in 
order  to  reflect  realistic  start  and  completion  dates  for  program  activities. 
Maintaining  the  integrity  of  the  schedule  logic  is  not  only  necessary  to 
reflect  true  status,  but  is  also  required  before  conducting  follow-on 
schedule  risk  analyses.  If  the  schedule  is  not  properly  updated,  positive 
and  negative  float  will  not  change  properly.  Positive  float  indicates  the 


51GAO-08-822  and  GAO-08-896. 

52See,  for  example,  GAO-09-3SP;  and  OMB  Capital  Programming  Guide  V  2.0,  Supplement 
to  Office  of  Management  and  Budget  Circular  A-ll,  Part  7:  Planning,  Budgeting,  and 
Acquisition  of  Capital  Assets  (Washington,  D.C.:  June  2006). 
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amount  of  time  the  schedule  can  fluctuate  before  affecting  the  end  date. 
Negative  float  indicates  critical  path  effort  that  may  require  management 
action  such  as  overtime,  second  or  third  shifts,  or  resequencing  of  work. 
Moreover,  if  activities  are  not  properly  sequenced  with  logical  links,  it  is 
not  certain  whether  the  critical  path — which  represents  the  chain  of 
dependent  activities  with  the  longest  total  duration — is  valid.  Table  4 
summarizes  the  results  of  our  review  of  the  four  programs. 


Table  4:  Extent  to  Which  Program  Schedules  Met  Best  Practices 

Extent  best  practice  met 

Best  practice 

DEAMS 

ECSS3 

GFEBS 

GCSS-Army 

1 .  Capturing  all  activities 

Partially 

Substantially 

Substantially 

Partially 

2.  Sequencing  all  activities 

Minimally 

Partially 

Partially 

Partially 

3.  Assigning  resources  to  all  activities 

Fully  met 

Minimally 

Not  Met 

Substantially 

4.  Establishing  the  duration  of  all  activities 

Substantially 

Substantially 

Fully  Met 

Fully  Met 

5.  Integrating  schedule  activities  horizontally  and  vertically 

Minimally 

Partially 

Minimally 

Partially 

6.  Establishing  the  critical  path  for  all  activities 

Minimally 

Partially 

Partially 

Partially 

7.  Identifying  reasonable  float  between  activities 

Minimally 

Partially 

Minimally 

Substantially 

8.  Conducting  a  schedule  risk  analysis 

Minimally 

Not  met 

Not  met 

Minimally 

9.  Updating  schedule  using  logic  and  durations  to  determine  dates 

Minimally 

Partially 

Partially 

Substantially 

Sources:  GAO  analysis  based  on  data  provided  by  the  PMOs. 

Note:  “Not  met”  means  the  program  provided  no  evidence  that  satisfies  any  of  the  criterion. 
“Minimally”  means  the  program  provided  evidence  that  satisfies  a  small  portion  of  the  criterion. 
"Partially”  means  the  program  provided  evidence  that  satisfies  about  half  of  the  criterion. 
“Substantially”  means  the  program  provided  evidence  that  satisfies  a  large  portion  of  the  criterion. 
“Fully  met”  means  the  program  provided  evidence  that  completely  satisfies  the  criterion. 

aln  reviewing  ECSS  we  analyzed  two  project  schedules:  (1)  solutions  development  and  (2)  reports, 
interfaces,  conversions,  and  extensions  (RICE).  We  analyzed  two  schedules  because  the  ECSS  IMS 
is  made  up  of  46  individual  project  schedules.  The  ratings  were  exactly  the  same  for  the  nine 
practices. 

Highlighted  below  are  examples  of  the  specific  weaknesses  we  found  in 
each  of  the  nine  best  practices.53  Appendix  IV  contains  a  detailed 
discussion  of  the  extent  to  which  the  four  ERPs  we  analyzed  met  the  nine 
best  practice  criteria. 

•  Capturing  all  activities.  A  schedule  should  reflect  all  activities  as  defined 
in  the  program’s  work  breakdown  structure  to  include  activities  to  be 
performed  by  the  government  and  the  contractor.  Our  analysis  found  that 


53GAO-09-3SP. 
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the  ERP  program  schedules  differed  in  the  extent  to  which  they  capture  all 
activities,  as  well  as  in  the  integration  of  government  and  contractor 
activities.  The  DEAMS  PMO  does  not  have  a  single  schedule  that 
integrates  government  and  contractor  activities.  While  the  PMO  maintains 
internal  schedules  that  reflect  government-only  activities,  these  activities 
are  not  linked  to  contractor  activities.  In  addition,  many  contractor 
activities  within  the  DEAMS  schedule  are  not  mapped  to  the  work 
breakdown  structure,  hampering  management’s  ability  to  ensure  all  effort 
is  included  in  the  schedule.  While  the  GCSS-Army  schedule  identifies 
contractor  activities,  it  contains  only  key  government  milestones  for  the 
program.  Other  government  activities,  such  as  testing  events  and 
milestones  beyond  December  2010,  are  not  captured  in  the  schedule.  The 
ECSS  program  schedule  contains  detailed  activities  associated  with 
government  effort  and  contractor  effort.  However,  the  government 
activities  are  not  fully  linked  to  contractor  activities,  so  that  updates  to 
government  activities  do  not  have  a  direct  impact  on  scheduled  contractor 
activities.  While  the  GFEBS’s  schedule  captures  government  and 
contractor  activities,  dependencies  between  key  milestones  in 
deployment,  software  release,  and  maintenance  are  not  linked,  thereby 
precluding  a  comprehensive  view  of  the  entire  program.  Without  fully 
integrating  government  activities  with  contractor  activities,  the  schedule 
will  not  be  able  to  reliably  estimate  the  date  the  program  is  to  be  finished 
if  a  significant  amount  of  key  activities  are  not  adequately  captured. 

•  Sequencing  all  activities.  The  schedule  should  be  planned  so  that  it  can 
meet  program  critical  dates.  To  meet  this  objective,  activities  need  to  be 
logically  sequenced  in  the  order  that  they  are  to  be  carried  out  and  no 
artificial  date  constraints  should  be  included  in  the  schedule.  In  particular, 
activities  that  must  finish  prior  to  the  start  of  follow-on  activities  (i.e., 
predecessor  activities),  as  well  as  activities  that  cannot  begin  until  other 
activities  are  completed  (i.e.,  successor  activities),  should  be  identified. 
None  of  the  contractor  schedules  we  assessed  fully  met  the  criteria  for 
sequencing  all  activities.  For  example,  the  DEAMS  schedule  has  over 
60  percent  of  the  remaining  activities  missing  logic  links  to  predecessor  or 
successor  activities.  Missing  predecessors  or  successors  reduce  the 
credibility  of  the  calculated  dates.  The  DEAMS  schedule  also  has  date 
constraints54  that  keep  the  schedule  from  responding  correctly  to  changes. 
The  ECSS  schedule  has  78  instances  of  unusual  logic  that  cause  activities 


54 A  constraint  predefines  the  start,  finish,  or  both  dates  of  an  activity.  The  schedule  should 
use  logic  and  durations  in  order  to  reflect  realistic  start  and  completion  dates  for  activities. 


Page  40 


GAO-11-53  DOD  Business  Systems 


to  finish  at  the  same  time  that  their  predecessor  activities  start.56  The 
GCSS-Army  schedule  has  constraints  on  1,503  of  the  remaining  activities 
that  keep  the  schedule  from  responding  to  changes.  Moreover,  the  GFEBS 
schedule  has  date  constraints  and  linked  summary  activities  that  interfere 
with  the  critical  path.56  Missing  or  incorrect  logic  reduces  the  credibility  of 
the  calculated  dates  in  the  schedule  because  the  schedule  will  not  reflect 
the  effects  of  slipping  activities  on  the  critical  path,  scheduled  resources, 
or  scheduled  start  dates  of  future  activities. 

•  Assigning  resources  to  all  activities.  The  schedule  should  realistically 
reflect  what  resources  (i.e.,  labor,  material,  and  overhead)  are  needed  to 
do  the  work,  whether  all  required  resources  will  be  available  when 
needed,  and  whether  any  funding  or  time  constraints  exist.  Because  of  the 
fixed  price  contractual  arrangements  with  ERP  contractors,  resources  are 
not  reflected  in  the  periodic  updates  of  the  schedules  submitted  to  the 
PMOs.  While  the  GCSS-Army  IMS  does  not  include  resources,  scheduled 
activities  can  be  traced  to  control  account  plans  which  have  resources  laid 
out  by  month  by  labor  category.  On  the  other  hand,  the  DEAMS  PMO 
provided  evidence  that  resources  were  assigned  to  activities  in  the 
schedule.  In  the  case  of  ECSS,  PMO  officials  stated  the  contractor 
assigned  resources  to  scheduled  activities,  but  we  were  not  able  to  verify 
whether  resources  were  assigned.  The  GFEBS  contractor’s  schedules  had 
resources  assigned  to  activities  in  earlier  releases  of  the  system,  but 
according  to  the  PMO,  resources  are  no  longer  assigned  to  activities. 
Without  resource  information,  DOD  management  has  insufficient  insight 
into  current  or  projected  over-allocation  of  contractor  resources,  thus 
increasing  the  risk  of  slippage  in  the  estimated  completion  date. 

•  Establishing  the  duration  of  all  activities.  The  schedule  should  reflect 
how  long  each  activity  will  take  to  execute  and  activity  durations  should 
be  as  short  as  possible  with  specific  start  and  end  dates.  The  four 
programs  properly  reflected  how  long  each  activity  should  take  to 
execute.  In  addition,  activities  were  generally  shorter  than  44  working 
days — or  2  working  months — which  represents  best  practices  for  activity 
durations. 


55These  unusual  links  are  known  as  start-to-finish  links,  and  they  are  rarely,  if  ever,  used  in 
scheduling.  Schedules  should  contain  a  predominance  of  finish-to-start  logical 
relationships  so  that  one  can  know  which  activities  must  finish  before  others  begin. 

56Summary  activities  summarize  the  effort  of  multiple  lower-level  tasks. 
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•  Integrating  schedule  activities  horizontally  and  vertically.  The  schedule 
should  be  integrated  horizontally  and  vertically.  Horizontal  integration 
means  that  the  schedule  links  the  products  and  outcomes  associated  with 
already-sequenced  activities.  Horizontal  integration  also  demonstrates  that 
the  overall  schedule  is  rational,  planned  in  a  logical  sequence  or  to  reflect 
interdependencies  between  work  and  planning  packages  and  provides  a 
way  to  evaluate  current  status.  When  schedules  are  vertically  integrated, 
lower-level  schedules  are  clearly  traced  to  upper-tiered  milestones, 
allowing  for  total  schedule  integration  and  enabling  different  teams  to 
work  to  the  same  schedule  expectations.  The  program  schedules  we 
assessed  partially  met  the  criteria  for  horizontal  and  vertical  integration.  In 
general,  as  discussed  earlier,  issues  with  missing  or  convoluted  logic  and 
artificially  constrained  dates  prevent  the  program  schedules  from  being 
horizontally  integrated.  Schedules  that  are  not  horizontally  integrated  may 
not  depict  relationships  between  different  program  elements  and  product 
handoffs.  While  ECSS  and  GCSS-Army  program  schedules  are  vertically 
integrated,  the  inability  to  clearly  trace  lower-level  schedules  to  upper¬ 
tiered  milestones  prevent  the  DEAMS  and  GFEBS  program  schedules  from 
being  fully  vertically  integrated. 

•  Establishing  the  critical  path.  The  establishment  of  a  critical  path — the 
longest  duration  path  through  the  sequenced  list  of  activities — is 
necessary  for  examining  the  effects  of  any  activity  slipping  along  this  path. 
The  calculation  of  a  critical  path  is  directly  related  to  the  logical 
sequencing  of  events.  Missing  or  convoluted  logic  and  artificially 
constrained  dates  prevent  the  calculation  of  a  valid  critical  path,  and  can 
mark  activities  as  critical  that  are  not  truly  critical.  The  program  schedules 
either  partially  or  minimally  met  the  criteria  for  establishing  a  critical  path. 
While  the  ECSS  PMO  has  insight  into  detailed  contractor  activities, 
officials  acknowledged  that  it  is  difficult  to  establish  a  critical  path  using 
the  program’s  current  schedule.  Instead,  the  program  tracks  high-level 
milestones  in  a  separate  schedule  and  officials  stated  that  they  are  in  the 
process  of  revamping  the  program’s  work  breakdown  structure  in  order  to 
establish  a  clearer  relationship  between  work  products  and  hence  a  more 
accurate  critical  path.  While  the  GFEBS  PMO  stated  that  it  receives 
weekly  updates  from  the  contractor  and  manages  to  a  critical  path,  our 
analysis  of  GFEBS  concluded  that  the  critical  path  was  not  reliable 
because  of  artificial  date  constraints  and  unrealistic  float.57  Conversely, 
GCSS-Army  and  DEAMS  officials  stated  that  regardless  of  insight  into 
detailed  contractor  activities,  a  critical  path  is  not  possible  because  it 


Float  is  the  amount  of  time  by  which  a  predecessor  activity  can  slip  before  the  delay 
affects  successor  activities. 
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would  be  too  complex.  However,  as  program  complexity  increases,  so 
must  the  schedule’s  sophistication.  Further,  our  analysis  of  the  DEAMS 
schedule  found  that  it  would  be  impossible  to  develop  a  critical  path 
because  60  percent  of  the  remaining  activities  are  missing  logic  links. 
Likewise,  our  analysis  found  that  a  critical  path  within  the  GCSS-Army 
schedule  is  not  possible  because  of  artificial  date  constraints  placed  on 
key  activities.  While  the  level  of  complexity  in  an  ERP  is  daunting,  a 
critical  path  through  at  least  a  higher-level  version  of  the  detailed  schedule 
would  assist  management  in  identifying  which  slipped  tasks  will  have 
detrimental  effects  on  the  completion  date.  By  managing  to  pre-defined, 
constrained  dates  instead  of  a  critical  path,  management  does  not  have  a 
clear  picture  of  the  tasks  that  must  be  performed  to  achieve  the  target 
completion  date. 

•  Identifying  reasonable  float  between  activities.  The  schedule  should 
identify  float — the  time  that  a  predecessor  activity  can  slip  before  the 
delay  affects  successor  activities — so  that  schedule  flexibility  can  be 
determined.  As  a  general  rule,  activities  along  the  critical  path  typically 
have  the  least  amount  of  float.  The  DEAMS,  ECSS,  and  GFEBS  schedules 
did  not  meet  the  criteria  for  identifying  reasonable  float.  The  missing  or 
convoluted  logic  and  artificially  constrained  dates  identified  above  prevent 
the  proper  calculation  of  float,  which  in  turn  affects  the  identification  of  a 
valid  critical  path.  Without  proper  insight  into  float,  management  cannot 
determine  the  flexibility  of  tasks  and  therefore  cannot  properly  reallocate 
resources  from  tasks  that  can  safely  slip  to  tasks  that  cannot  slip  without 
adversely  affecting  the  estimated  program  completion  date. 

•  Conducting  a  schedule  risk  analysis.  A  schedule  risk  analysis  uses 
statistical  techniques  to  predict  a  level  of  confidence  in  meeting  a 
completion  date.  The  purpose  of  the  analysis  is  to  develop  a  probability 
distribution  of  possible  completion  dates  that  reflect  the  project  and  its 
quantified  risks.  This  analysis  can  help  management  to  understand  the 
most  important  risks  and  to  focus  on  mitigating  these  risks.  We  found  that 
none  of  the  PMOs  have  explicitly  linked  program  risks  to  their  schedule  in 
the  form  of  a  schedule  risk  analysis.  The  ECSS  PMO  stated  that  it  actively 
monitors  schedule  risk,  but  it  has  not  performed  a  schedule  risk  analysis. 
The  GFEBS  PMO  stated  that  while  schedule  risks  have  been  discussed  in 
team  meetings,  it  has  not  performed  a  formal  schedule  risk  analysis. 
However,  the  GFEBS  PMO  stated  that  it  is  open  to  improving  in  the  area  of 
schedule  risk  analysis.  The  DEAMS  PMO  stated  that  while  it  has  tied  risks 
to  activities  the  program  considers  to  be  on  the  critical  path,  a  formal 
schedule  risk  analysis  was  not  performed  because  the  schedule  provided 
by  the  contractor  lacks  a  sufficient  level  of  detail  to  do  such  an  analysis. 
The  GCSS-Army  contractor  recently  conducted  a  high-level  schedule  risk 
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analysis  on  two  major  milestones.  In  addition,  GCSS-Army  PMO  officials 
acknowledged  the  importance  of  a  detailed  schedule  risk  analysis  and 
stated  that  the  PMO  intends  to  include  this  requirement  in  the  contract 
within  the  next  few  months.  A  schedule  risk  analysis  is  important  because 
it  allows  high-priority  risks  to  be  identified  and  mitigated,  and  the  level  of 
confidence  in  meeting  projected  completion  dates  can  be  predicted. 
Without  a  schedule  risk  analysis,  the  PMO  cannot  reliably  determine  the 
level  of  confidence  in  meeting  the  completion  date.  However,  if  the 
schedule  risk  analysis  is  to  be  credible,  the  program  must  have  a  quality 
schedule  that  reflects  reliable  logic  and  clearly  identifies  the  critical 
path — conditions  that  none  of  the  ERP  schedules  met. 

•  Updating  the  schedule  using  logic  and  durations  to  determine  dates.  The 
schedule  should  use  logic  and  durations  in  order  to  reflect  realistic  start 
and  completion  dates.  The  schedule  should  be  continually  monitored  to 
determine  when  forecasted  completion  dates  differ  from  the  planned 
dates,  which  can  be  used  to  determine  whether  schedule  variances  will 
affect  future  work.  There  are  differences  in  the  four  programs’  ability  to 
update  the  schedule  using  logic  and  durations  to  determine  schedule 
dates.  For  example,  the  GCSS-Army  schedule  substantially  met  the  criteria 
for  updating  the  schedule,  but  the  GFEBS  schedule  has  over  100  instances 
of  activities  that  should  have  occurred,  yet  have  no  actual  start  dates  or 
finish  dates.  In  addition,  the  ECSS  schedules  had  a  status  date  of 
January  1,  2010 — a  federal  holiday — while  schedules  provided  to  the 
DEAMS  program  office  by  the  contractor  did  not  have  a  status  date.  A 
status  date  denotes  the  date  of  the  latest  update  to  the  schedule  and 
therefore  defines  the  point  in  time  at  which  completed  work  and 
remaining  work  are  calculated.  Without  a  valid  status  date,  management  is 
not  able  to  determine  what  work  is  completed  and  what  work  is 
remaining.  An  invalid  or  missing  status  date  is  also  an  indication  that 
management  is  not  using  the  schedule  to  effectively  oversee  and  monitor 
the  effort.  Furthermore,  maintaining  the  integrity  of  the  schedule  logic  is 
not  only  necessary  to  reflect  true  status,  but  is  also  required  before 
conducting  a  schedule  risk  analysis. 

Each  of  the  PMOs  acknowledged  the  importance  of  many  of  the 
scheduling  best  practices,  but  stated  that  its  ability  to  meet  the  prescribed 
best  practices  is  limited  because  of  the  complexity  of  the  ERP 
development  process  and  the  use  of  the  firm-fixed  price  contract.  Under 
the  terms  of  these  firm-fixed  price  contracts,  the  contractors  are  not 
required  to  provide  detailed  and  timely  scheduling  data,  which  are 
essential  for  preparing  order  and  an  accurate  and  reliable  schedule  for  the 
implementation  of  the  system.  While  some  of  the  necessary  information  is 
not  being  provided  by  the  contractor,  this  does  not  relieve  the  PMOs  of  the 
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responsibility  for  developing  an  IMS  that  fully  meets  prescribed  best 
practices.  Without  the  development  of  an  IMS  that  meets  scheduling  best 
practices,  the  PMOs  and  the  department  are  not  positioned  to  adequately 
monitor  and  oversee  the  progress  of  the  billions  of  dollars  being  invested 
in  the  modernization  of  DOD’s  business  systems.  Lacking  a  credible  IMS, 
management  is  unable  to  predict,  with  any  degree  of  confidence,  whether 
the  estimated  completion  date  is  realistic.  An  integrated  schedule  is  key  in 
managing  program  performance  and  is  necessary  for  determining  what 
work  remains  and  the  expected  cost  to  complete  it.  A  schedule  delay  can 
also  lead  to  an  increase  in  the  cost  of  the  project  because,  for  example, 
labor,  supervision,  facilities,  and  escalation  cost  more  if  the  program  takes 
longer.  A  schedule  and  cost  risk  assessment  recognizes  the 
interrelationship  between  schedule  and  cost  and  captures  the  risk  that 
schedule  durations  and  cost  estimates  may  vary.  But  without  a  fully 
integrated  master  schedule,  the  full  extent  of  schedule  uncertainty  is  not 
known,  and  therefore  cannot  be  incorporated  into  the  cost  uncertainty 
analysis  as  schedule  risk. 

Subsequent  to  the  completion  of  our  field  work,  ECSS  and  GFEBS  PMOs 
provided  updated  schedules  for  assessment,  but  we  were  unable  to 
perform  a  detailed  evaluation  of  the  updated  schedules.  Program  officials 
for  ECSS  and  GFEBS  indicated  that  the  updated  schedules  addressed 
some  areas  in  which  their  previous  schedules  were  deficient  according  to 
GAO’s  assessment  of  the  nine  scheduling  best  practices.  In  response  to 
limitations  that  we  identified  and  shared  with  the  GFEBS  PMO,  the 
program  office  enacted  several  formal  changes  to  its  existing  schedule. 
The  ECSS  PMO  provided  us  with  an  updated  IMS  that  contains  details  on 
future  activities  beyond  the  scheduled  activities  we  originally  assessed. 
Although  we  did  not  assess  the  new  schedule,  according  to  the  ECSS 
PMO,  the  updated  schedule  is  an  improvement  over  past  versions  of  the 
ECSS  schedule  and  addresses  many  of  the  deficiencies  GAO  identified  in 
the  earlier  version. 
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Although  Cost  Estimates 
Meet  Most  Best  Practices, 
the  Lack  of  Sensitivity  and 
Uncertainty  Analyses 
Results  in  Estimates  That 
May  Not  Be  Credible 


We  have  identified58  four  characteristics  of  a  reliable  cost  estimate 
(1)  well-documented,  (2)  comprehensive,  (3)  accurate,  and  (4)  credible. 
The  four  characteristics  encompass  12  best  practices  for  effective  program 
cost  estimates  that  are  identified  in  appendix  V.  The  results  of  our  review 
of  the  DEAMS,  ECSS,  GFEBS,  and  GCSS-Army  cost  estimates  are 
summarized  in  table  5. 


Table  5:  Extent  Cost  Estimates  Met  Best  Practices 


Best  practice 

DEAMS 

ECSS 

GFEBS 

GCSS-Army 

Well-documented 

Substantially 

Substantially 

Fully  met 

Substantially 

Comprehensive 

Fully  met 

Fully  met 

Fully  met 

Substantially 

Accurate 

Fully  met 

Substantially 

Substantially 

Partially 

Credible 

Fully  met 

Partially 

Minimally 

Partially 

Sources:  GAO  analysis  based  on  information  provided  by  the  PMOs. 

Note:  “Not  met”  means  the  program  provided  no  evidence  that  satisfies  any  of  the  criterion. 
“Minimally”  means  the  program  provided  evidence  that  satisfies  a  small  portion  of  the  criterion. 
“Partially”  means  the  program  provided  evidence  that  satisfies  about  half  of  the  criterion. 
“Substantially”  means  the  program  provided  evidence  that  satisfies  a  large  portion  of  the  criterion. 
“Fully  met”  means  the  program  provided  evidence  that  completely  satisfies  the  criterion. 

Highlighted  below  are  examples  of  the  specific  strengths  and  weaknesses 
we  found  in  each  of  the  four  best  practices.  Appendix  V  contains  a 
detailed  discussion  of  the  extent  to  which  the  four  ERPs  we  analyzed  met 
the  four  best  practices  criteria. 

•  Well-documented.  The  cost  estimates  should  be  supported  by  detailed 
documentation  that  describes  the  purpose  of  the  estimate,  the  program 
background  and  system  description,  the  scope  of  the  estimate,  the  ground 
rules  and  assumptions,  all  data  sources,  estimating  methodology  and 
rationale,  and  the  results  of  the  risk  analysis.  Moreover,  this  information 
should  be  captured  in  such  a  way  that  the  data  used  to  derive  the  estimate 
can  be  traced  back  to,  and  verified  against,  their  sources.  The  cost 
estimates  for  DEAMS,  ECSS,  GFEBS,  and  GCSS-Army  are  well- 
documented.  The  cost  estimates  have  clearly  defined  purposes  and  are 
supported  by  documented  descriptions  of  key  program  or  system 
characteristics  (e.g.,  relationships  with  other  systems,  performance 
parameters).  Additionally,  they  capture  in  writing  such  things  as  the 


58GAO-09-3SP. 
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source  data  used  and  their  significance,  the  calculations  performed  and 
their  results,  and  the  rationale  for  choosing  a  particular  estimating  method 
or  reference.  This  information  is  captured  in  such  a  way  that  the  data  used 
to  derive  the  estimate  can  be  traced  back  to,  and  verified  against,  the 
sources.  The  final  cost  estimates  are  reviewed  and  accepted  by 
management  on  the  basis  of  confidence  in  the  estimating  process  and  the 
estimate  produced  by  the  process. 

•  Comprehensive.  The  cost  estimates  should  include  costs  of  the  program 
over  its  full  life  cycle,  provide  a  level  of  detail  appropriate  to  ensure  that 
cost  elements  are  neither  omitted  nor  double-counted,  and  document  all 
cost-influencing  ground  rules  and  assumptions.  We  found  that  cost 
estimates  for  DEAMS,  ECSS,  GFEBS,  and  GCSS-Army  are  comprehensive. 
The  cost  estimates  include  both  government  and  contractor  costs  over  the 
program’s  life  cycle,  from  the  inception  of  the  program  through  design, 
development,  deployment,  and  operation  and  maintenance  to  retirement. 
They  also  provide  an  appropriate  level  of  detail  to  ensure  that  cost 
elements  are  neither  omitted  nor  duplicated  and  include  documentation  of 
all  cost-influencing  ground  rules  and  assumptions. 

•  Accurate.  The  cost  estimates  should  be  based  on  an  assessment  of  most 
likely  costs  (adjusted  for  inflation),  documented  assumptions,  and 
historical  cost  estimates  and  actual  experiences  on  other  comparable 
programs.  Estimates  should  be  cross-checked  against  an  independent  cost 
estimate  for  accuracy,  double  counting,  and  omissions.  In  addition,  the 
estimates  should  be  updated  to  reflect  any  changes.  Our  analysis  also 
found  the  cost  estimates  for  DEAMS,  ECSS,  and  GFEBS  to  be  accurate. 
The  cost  estimates  provide  for  results  that  are  unbiased  and  are  not  overly 
conservative  or  optimistic.  In  addition,  the  cost  estimates  are  updated 
regularly  to  reflect  material  changes  in  the  program,  and  steps  are  taken  to 
minimize  mathematical  mistakes  and  their  significance.  Among  other 
things,  the  cost  estimates  are  grounded  in  historical  record  of  cost 
estimating  and  actual  experiences  on  comparable  programs.  Our  analysis 
found  the  cost  estimate  for  GCSS-Army  to  be  partially  accurate  because 
we  could  not  verify  how  actual  incurred  costs  were  used  to  update  the 
cost  estimate. 

•  Credible.  The  cost  estimates  should  discuss  any  limitations  of  the  analysis 
because  of  uncertainty,  or  biases  surrounding  data  or  assumptions.  Risk 
and  uncertainty  analysis  should  be  performed  to  determine  the  level  of 
risk  associated  with  the  estimate.  Further,  the  estimate’s  results  should  be 
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cross-checked  against  an  independent  cost  estimate.69  While  we  found  that 
the  ERP  programs  were  generally  following  the  cost  estimating  best 
practices,  our  analysis  also  found  that  the  cost  estimates  for  ECSS, 

GFEBS,  and  GCSS-Army  are  not  fully  credible.  As  stipulated  in  OMB  and 
DOD  guidance,  ECSS  and  GCSS-Army  did  not  include  a  sensitivity 
analysis,  and  GFEBS  did  not  include  a  sensitivity  analysis  or  a  cost  risk 
and  uncertainty  analysis.  In  the  case  of  GFEBS,  our  July  2007  report60 
noted  that  a  sensitivity  analysis  had  not  been  developed  in  calculating  the 
life-cycle  cost  estimate.  Cost  estimates  should  discuss  any  limitations  of 
the  analysis  because  of  uncertainty  or  biases  surrounding  data  and 
assumptions.  Major  assumptions  should  be  varied,  and  other  outcomes 
recomputed  to  determine  how  sensitive  they  are  to  changes  in  the 
assumptions.  Having  a  range  of  costs  around  a  point  estimate  is  more 
useful  to  decision  makers  because  it  conveys  the  level  of  confidence  in 
achieving  the  most  likely  cost  and  also  informs  them  of  cost,  schedule,  and 
technical  risks.  In  addition,  as  discussed  earlier,  because  each  of  the  four 
programs  we  assessed  did  not  meet  best  practices  for  schedule  estimating, 
none  of  the  cost  estimates  could  be  considered  credible  because  they  did 
not  assess  the  cost  effects  of  schedule  slippage.  While  individual  phases  of 
a  multi-phased  project  may  be  completed  on  time,  the  project  as  a  whole 
can  be  delayed,  and  phases  that  are  not  part  of  an  IMS  may  not  be 
completed  efficiently  which  could  result  in  future  cost  overruns. 

A  reliable  cost  estimate  is  a  key  variable  in  calculating  return  on 
investment,  and  it  provides  the  basis  for  informed  investment  decision 
making,  realistic  budget  formulation  and  program  resourcing,  meaningful 
progress  measurement,  proactive  course  correction,  and  accountability  for 
results.  According  to  OMB61  programs  must  maintain  current  and  well- 
documented  cost  estimates,  and  these  estimates  must  encompass  the  full 
life  cycle  of  the  program.  OMB  states  that  generating  reliable  cost 
estimates  is  a  critical  function  necessary  to  support  OMB’s  capital 
programming  process.  Without  reliable  estimates,  programs  are  at 


o9An  independent  cost  estimate  is  another  estimate  based  on  the  same  technical 
information  that  is  used  to  validate  and  cross-check  the  baseline  estimate,  but  is  prepared 
by  a  person  or  organization  that  has  no  stake  in  the  approval  of  the  project. 

60GAO-07-860. 

61OMB  Circular  No.  A-ll,  Preparation,  Submission,  and  Execution  of  the  Budget  (June 
2006);  OMB  Circular  No.  A-130,  Revised,  Management  of  Federal  Information  Resources 
(Nov.  28,  2000);  and  Office  of  Management  and  Budget,  Capital  Programming  Guide: 
Supplemeyit  to  Circular  A- 1 1 ,  Part  7,  Preparation,  Submission,  and  Execution  of  the 
Budget  (June  2000). 
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increased  risk  of  experiencing  cost  increases,  missed  deadlines,  and 
performance  shortfalls. 


ERP  Success  in 
Transforming 
Business  Operations 
Has  Not  Been  Defined 
or  Measured 


DOD  has  not  yet  defined  success  for  ERP  implementation  in  the  context  of 
business  operations  and  in  a  way  that  is  measurable.  Accepted  practices  in 
system  development  include  testing  the  system  in  terms  of  the 
organization’s  mission  and  operations — whether  the  system  performs  as 
envisioned  at  expected  levels  of  cost  and  risk  when  implemented  within 
the  organization’s  business  operations.  The  Clinger-Cohen  Act  of  1996 
recognizes  the  importance  of  performance  measurement  in  requiring 
agencies  to  (1)  establish  goals  for  improving  the  efficiency  and 
effectiveness  of  agency  operations  and  (2)  ensure  that  performance 
measurements  determine  how  well  the  information  technology  supports 
programs  of  the  executive  agency.62 


DOD  also  has  recognized  the  importance  of  performance  measures,  which 
the  department  directs  should  be  (1)  written  in  terms  of  desired  outcomes, 

(2)  quantifiable,  (3)  able  to  measure  the  degree  to  which  the  desired 
outcome  is  achieved,  (4)  independent  of  the  particular  automated  system 
tested  and  not  focused  on  system-performance  criteria,  and  (5)  designed 
to  include  benefits  to  the  DOD  component  and  the  enterprise.  In  regard  to 
the  ERPs,  measures  determining  whether  the  system  is  being  used  as 
expected  and  is  providing  the  desired  benefits  from  a  business  perspective 
will  vary  depending  on  the  specific  type  of  business  functions  the  system 
is  performing.  For  example,  in  a  logistical  system,  the  new  system  and  its 
processes  may  be  expected  to  help  accomplish  such  items  as  (1)  reducing 
inventory  levels;  (2)  increasing  the  inventory  turnover  rate  which  shows 
that  the  items  actually  needed  are  the  items  being  procured;  and 

(3)  increasing  the  accuracy  of  the  projected  completion  dates  for  repair 
projects  which  allows  for  better  equipment  utilization.  On  the  other  hand, 
a  financial  system  may  measure  benefits  in  such  areas  as  (1)  reducing 
prompt  payment  penalties;  (2)  improving  the  financial  statement 
preparation  process  by  having  the  system  automatically  generate  the 
statements  which  reduces  the  potential  for  manual  error;  and 

(3)  improving  management  oversight  of  an  entity’s  operations  and 
providing  the  detailed  data  necessary  to  evaluate  abnormalities  that  may 
be  detected.  Developing  and  using  specific  performance  measures  to 


62Pub.  L.  No.  104-106,  div.  E,  title  LI,  §  6123,  110  Stat.  679,  683-84  (Feb.  10,  1996),  codified, 
as  amended,  at  40  U.S.C.  §  11313. 
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evaluate  a  system  effort  should  help  management  understand  whether  the 
expected  benefits  are  being  realized. 

While  the  definition  of  success  and  performance  measures  for  DOD’s 
ERPs  will  differ  between  organizational  levels,  components,  and 
subcomponents,  our  previous  work  has  shown  that  performance  measures 
should  be  aligned  toward  a  shared  direction.  In  this  regard,  all  members  of 
the  organization  need  to  understand  the  ultimate  result  to  be  achieved  and 
all  parties  should  work  toward  the  same  goal  and  desired  results.  This 
alignment  should  extend  throughout  the  organization  and  cover  the 
activities  that  an  entity  is  expected  to  perform  to  support  the  intent  of  the 
program.63  DOD  has  not  taken  actions  to  align  the  definitions  and  related 
performance  measures  used  by  its  components  to  measure  progress  and 
determine  success. 

The  DCMOs  told  us  that  they  had  not  yet  developed  a  DOD-wide  definition 
of  success  or  related  performance  measures  for  ERPs.  While 
acknowledging  the  importance  of  these  practices,  the  officials  told  us  that 
they  are  still  in  the  early  stages  of  implementing  processes  for  managing 
and  overseeing  their  business  systems  modernization  efforts,  in 
accordance  with  the  fiscal  year  2010  Defense  Authorization  Act. 

Successful  implementation  of  the  ERPs  is  critical  to  transforming  business 
operations.  Without  defining  ERP  success  in  terms  of  support  for  mission 
and  business  operations  and  establishing  the  related  performance 
measures,  the  military  services  and  the  department  cannot  ensure  that  the 
performance  of  deployed  ERPs  has  been  realistically  and  accurately 
measured. 

Our  April  2010  report,64  which  focused  on  the  second  deployment  of  LMP 
at  the  Corpus  Christi  and  Letterkenny  Army  Depots,  illustrates  the 
importance  of  establishing  performance  measures.  Based  on  our 
observations  at  the  Corpus  Christi  and  Letterkenny  Army  Depots,  we 
found  that  the  Army’s  measures  for  assessing  LMP  implementation  at  the 
two  deployment  sites  did  not  accurately  reflect  whether  the  locations  were 
able  to  perform  their  day-to-day  operations  using  LMP  as  envisioned. 
Rather,  the  measures  used  by  the  Army  assessed  the  success  at  the  two 
locations  from  a  system-software  perspective.  While  this  is  important, 


63GAO,  Tax  Administration:  IRS  Needs  to  Further  Refine  Its  Tax  Filing  Season 
Performance  Measures,  GAO-03-143  (Washington,  D.C.:  Nov.  22,  2002). 

04GAO-1O-461. 
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performance  measures  from  a  business  perspective  were  not  considered 
to  determine  whether  the  depots  were  able  to  use  LMP  to  perform  their 
mission  to  repair  items.  Without  performance  measures  to  evaluate  how 
well  these  systems  are  accomplishing  their  desired  goals,  DOD  decision 
makers  including  program  managers  do  not  have  all  the  information  they 
need  to  evaluate  their  investments  to  determine  whether  the  individual 
programs  are  helping  DOD  achieve  business  transformation  and  thereby 
improve  upon  its  primary  mission  of  supporting  the  warfighter. 


Conclusions 


Modernizing  the  department’s  business  systems  is  a  critical  part  of 
transforming  DOD’s  business  operations,  addressing  some  of  its  high-risk 
areas,  and  providing  more  accurate  and  reliable  financial  information  to 
the  Congress  on  the  results  of  DOD’s  operations.  However,  DOD  continues 
to  experience  difficulties  that  hinder  its  ability  to  implement  these  efforts 
on  time  and  within  budget.  The  department  has  not  followed  best 
practices  and  developed  a  reliable  IMS  for  several  of  these  modernization 
efforts.  As  a  result,  it  lacks  the  assurance  that  these  ERPs  will  be 
completed  by  the  projected  date.  Furthermore,  while  DOD  generally 
followed  best  practices  in  developing  the  programs’  cost  estimates  for 
these  efforts,  with  the  exception  of  DEAMS,  none  of  the  programs  has 
prepared  a  sensitivity  analysis.  The  lack  of  a  sensitivity  analysis  increases 
the  chances  that  decisions  will  be  made  without  a  clear  understanding  of 
the  possible  impact  on  the  estimates  of  costs  and  benefits  of  each 
program.  In  addition,  because  each  of  the  four  programs  we  assessed  did 
not  meet  best  practices  for  schedule  estimating,  none  of  the  cost  estimates 
could  be  considered  credible  because  they  did  not  assess  the  cost  effects 
of  schedule  slippage.  It  is  critical  to  correct  the  underlying  issues  to  help 
ensure  that  the  billions  of  dollars  spent  annually  are  being  used  in  the 
most  efficient  and  effective  manner.  While  modernizing  its  business 
systems  is  not  a  risk-free  endeavor,  additional  funds  spent  because  of 
schedule  slippages  are  funds  that  could  have  been  available  for  other 
departmental  priorities.  Furthermore,  the  longer  it  takes  to  implement 
these  critical  business  systems,  the  longer  the  department  will  continue  to 
use  its  existing  duplicative,  stovepiped  systems  environment  and  further 
erode  the  estimated  savings  that  were  to  accrue  to  DOD  as  a  result  of 
modernizing  its  business  systems. 

Additionally,  the  department  has  not  defined  the  measures  to  ascertain  if 
the  systems  are  providing  the  desired  functionality  to  achieve  DOD’s 
business  transformation  goals.  DOD  has  stated  that  the  successful 
implementation  of  the  ERPs  is  critical  to  transforming  its  business 
operations  and  addressing  some  of  its  high-risk  areas.  However,  we  found 
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that  the  department  has  not  yet  developed  performance  measures  to 
ascertain  whether  the  systems,  once  implemented,  are  providing  the 
intended  functionality.  If  the  systems  cannot  be  used  for  their  intended 
purpose,  transformation  will  be  difficult  if  not  impossible  to  achieve  and 
the  billions  of  dollars  being  invested  in  these  systems  may  not  generate  the 
benefits  and  efficiencies  as  intended.  Further,  we  reaffirm  our  prior 
recommendations  related  to  the  actions  needed  to  improve  the 
department’s  management  and  oversight  of  the  ERPs. 


Recommendations  for  strenst^en  DOD’s  management  oversight  and  accountability  over 

business  system  investments  and  help  provide  for  the  successful 
Executive  Action  implementation  of  the  ERPs,  we  recommend  that  the  Secretary  of  Defense 

take  the  following  eight  actions: 

•  Direct  the  Secretary  of  the  Army  to  ensure  that  the  Chief  Management 
Officer  of  the  Army  directs  the  PMO  for  the  GFEBS  to  develop  an  IMS  that 
fully  incorporates  best  practices.  The  schedule  should 

•  sequence  all  activities, 

•  assign  resources  to  all  activities, 

•  integrate  schedule  activities  horizontally  and  vertically, 

•  establish  the  critical  path  for  all  activities, 

•  identify  float  between  activities, 

•  conduct  a  schedule  risk  analysis,  and 

•  update  schedule  using  logic  and  durations  to  determine  dates. 

•  Direct  the  Secretary  of  the  Army  to  ensure  that  the  Chief  Management 
Officer  of  the  Army  direct  the  PMO  for  GCSS-Army  to  develop  an  IMS  that 
fully  incorporates  best  practices.  The  schedule  should 

•  capture  all  activities, 

•  sequence  all  activities, 

•  integrate  schedule  activities  horizontally  and  vertically, 

•  establish  the  critical  path  for  all  activities,  and 

•  conduct  a  schedule  risk  analysis. 


•  Direct  the  Secretary  of  the  Air  Force  to  ensure  that  the  Chief  Management 
Officer  of  the  Air  Force  directs  the  PMO  for  DEAMS  to  develop  an  IMS 
that  fully  incorporates  best  practices.  The  schedule  should 

•  capture  all  activities, 

•  sequence  all  activities, 

•  integrate  schedule  activities  horizontally  and  vertically, 
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•  establish  the  critical  path  for  ah  activities, 

•  identify  float  between  activities, 

•  conduct  a  schedule  risk  analysis,  and 

•  update  schedule  using  logic  and  durations  to  determine  dates. 

•  Direct  the  Secretary  of  the  Air  Force  to  ensure  that  the  Chief  Management 
Officer  of  the  Air  Force  directs  the  PMO  for  ECSS  to  develop  an  IMS  that 
fully  incorporates  best  practices.  The  schedule  should 

•  sequence  ah  activities, 

•  assign  resources  to  ah  activities, 

•  integrate  schedule  activities  horizontally  and  vertically, 

•  establish  the  critical  path  for  ah  activities, 

•  identify  float  between  activities, 

•  conduct  a  schedule  risk  analysis,  and 

•  update  schedule  using  logic  and  durations  to  determine  dates. 

•  Direct  the  Secretary  of  the  Army  to  ensure  that  the  Chief  Management 
Officer  of  the  Army  directs  the  PMO  for  GFEBS  to  update  the  cost 
estimates  by  preparing  sensitivity  and  risk  and  uncertainty  analyses  using 
best  practices. 

•  Direct  the  Secretary  of  the  Army  to  ensure  that  the  Chief  Management 
Officer  of  the  Army  directs  the  PMO  for  GCSS-Army  to  update  the  cost 
estimates  by  using  actual  cost  and  preparing  a  sensitivity  analysis  using 
best  practices. 

•  Direct  the  Secretary  of  the  Air  Force  to  ensure  that  the  Chief  Management 
Officer  of  the  Air  Force  directs  the  PMO  for  ECSS  to  update  the  cost 
estimates  by  preparing  a  sensitivity  analysis  using  best  practices. 

•  Direct  the  department’s  Chief  Management  Officer  and  the  chief 
management  officers  of  the  military  departments  to  establish  performance 
measures  based  on  quantitative  data  that  will  enable  the  department  to 
assess  whether  each  respective  military  service’s  ERP  efforts  are 
providing  the  intended  business  capabilities  to  the  system  users. 


Agency  Comments 
and  Our  Evaluation 


DOD  provided  written  comments  on  a  draft  of  this  report.  In  its  comments, 
DOD  concurred  with  the  eight  recommendations  and  cited  actions 
planned  to  address  them.  For  example,  the  department  recognized  the 
importance  of  an  integrated  master  schedule  as  a  key  program 
management  tool  fundamental  to  having  a  reliable  program  schedule.  The 
department  stated  that  the  appropriate  military  department  Chief 
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Management  Officer  will  direct  program  managers  to  implement  the 
recommendations  and  further  noted  that  the  Chief  Management  Officer 
will  oversee  the  implementation  of  the  recommendations.  Further,  DOD 
stated  that  guidance  will  be  issued  requiring  DOD  business  systems 
investments  to  include  performance  measures  that  can  be  used  to  assess 
the  expected  benefits  of  the  investments.  Additionally,  the  department 
noted  that  the  performance  measures  will  be  incorporated  into  DOD’s 
Business  Enterprise  Architecture. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense;  the 
Secretary  of  the  Army;  the  Secretary  of  the  Navy;  the  Secretary  of  the  Air 
Force;  the  Deputy  Secretary  of  Defense;  the  Under  Secretary  of  Defense 
(Comptroller);  the  Chief  Management  Officer  of  the  Army,  the  Navy,  and 
the  Air  Force;  the  program  management  office  for  each  business  system 
that  was  included  in  the  audit;  and  other  interested  congressional 
committees  and  members.  This  report  also  is  available  at  no  charge  on  the 
GAO  Web  site  at  http://www.gao.gov. 

Please  contact  Asif  A.  Khan  at  (202)  512-9095  or  khana@gao.gov  or 
Nabajyoti  Barkakati  at  (202)  512-4499  or  barkakatin@gao.gov  if  you  or 
your  staff  have  questions  on  matters  discussed  in  this  report.  Contact 
points  for  our  Offices  of  Congressional  Relations  and  Public  Affairs  may 
be  found  on  the  last  page  of  this  report.  Key  contributors  to  this  report  are 
listed  in  appendix  VI. 


Director 
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Appendix  I:  Objective,  Scope,  and 
Methodology 


Our  objectives  were  to  (1)  provide  the  status  as  of  December  31,  2009  of 
the  nine  enterprise  resource  planning  (ERP)  systems  that  the  Department 
of  Defense  (DOD)  identified  as  essential  to  transforming  its  business 
operations;  (2)  assess  the  scheduling  and  cost  estimating  practices  of 
selected  ERPs  to  determine  the  extent  to  which  the  program  management 
offices  (PMO)  were  applying  best  practices;  and  (3)  ascertain  whether 
DOD  and  the  military  departments  have  defined  the  performance 
measures  to  determine  whether  the  systems  will  meet  their  intended 
business  capabilities. 

To  address  the  first  objective,  we  obtained  and  reviewed  information 
provided  by  the  PMO  responsible  for  the  nine  ERP  efforts.1  More 
specifically,  we  obtained  data  related  to  the  following  for  each  program: 

(1)  when  the  program  was  initiated,  (2)  the  program’s  accountable  official, 
(3)  the  purpose  of  the  program,  (4)  the  cost  of  the  program,  (5)  the 
implementation  schedule,  (6)  the  number  of  legacy  systems  intended  to  be 
replaced,  (7)  the  cost  of  the  legacy  systems,  (8)  the  date  the  program  was 
last  certified,  and  (9)  the  conditions  placed  on  the  program  by  the  various 
review  boards.  For  the  purposes  of  this  report,  we  did  not  include 
information  on  the  Defense  Logistics  Agency  Business  System 
Modernization/Enterprise  Business  System.  According  to  DOD  the 
Business  System  Modernization  effort  was  fully  implemented  in  July  2007 
and  transformed  how  the  agency  conducts  its  operations  in  five  core 
business  processes:  order  fulfillment,  demand  and  supply  planning, 
procurement,  technical/quality  assurance,  and  financial  management. 
Subsequently,  in  September  2007,  the  name  of  the  program  was  changed  to 
the  Enterprise  Business  System,  which  is  a  continuation  of  the  ERP’s 
capabilities  to  support  internal  agency  operations. 

We  also  reviewed  various  DOD  documents  such  as  the  Enterprise 
Transition  Plans  issued  in  September  2008  and  December  2009,  the 
Defense  Business  Systems  Management  Committee  meeting  minutes  and 
briefings,  the  Selected  Capital  Investment  Reports,  which  are  prepared  in 
support  of  the  funding  requests  for  the  ERPs,  the  Congressional  Report  on 
Defense  Business  Operations  for  fiscal  years  2009  and  2010  and  the  Major 
Automated  Information  System  Reports  for  fiscal  years  2008  and  2009 — to 
corroborate  the  information  obtained  from  the  PMOs.  In  instances  where 


’This  engagement  focused  on  nine  ERP  efforts  that  DOD  considers  critical  to  transforming 
its  business  operations  and  resolving  some  of  the  department’s  high-risk  areas  such  as 
business  transformation,  business  system  modernization,  financial  management,  and 
supply  chain  management. 
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we  identified  discrepancies,  we  followed  up  with  the  PMOs  to  obtain  an 
explanation.  Most  of  the  financial  information  in  this  report  was  obtained 
through  interviews  with  or  responses  to  GAO  questions  from 
knowledgeable  PMO  officials  for  the  nine  ERP  systems.  As  part  of  the  first 
objective,  we  also  reviewed  past  GAO  reports2  that  were  specific  to  the 
department’s  efforts  to  implement  the  nine  ERPs  to  identify  prior 
recommendations  and  assess  DOD’s  progress  in  addressing  the 
19  recommendations  discussed  in  these  reports. 

To  assess  the  scheduling  and  cost  estimating  practices  of  selected  ERPs, 
we  selected  the  General  Fund  Enterprise  Business  System  (GFEBS),  the 
Global  Combat  Support  System-Army  (GCSS-Army),  the  Defense 
Enterprise  Accounting  and  Management  Systems  (DEAMS),  and  the 
Expeditionary  Combat  Support  System  (ECSS).  The  other  programs  were 
excluded  because  (1)  the  Logistics  Modernization  Program  is  expected  to 
be  fully  deployed  soon,  (2)  it  is  too  soon  to  assess  the  department’s 
integrated  personnel  and  pay  efforts  because  of  the  recent  change  in  the 
Defense  Integrated  Military  Human  Resources  System  implementation 
strategy  and  the  Defense  Agencies  Initiative  has  yet  to  develop  its 
implementation  schedule  for  the  various  defense  agencies,  and  (3)  we 
reported3  on  concerns  with  the  Marine  Corps  and  Navy  schedule  and  cost 
estimating  practices  in  July  2008  and  September  2008,  respectively.  In 
performing  our  analysis  for  the  four  ERPs,  we  reviewed  the  schedules  and 
cost  estimates  available  at  the  time  of  our  review  and  evaluated  them 
using  the  criteria  set  forth  in  GAO’s  cost  guide.4 5  In  using  the  guide,  we 
determined  the  extent  to  which  each  schedule  was  prepared  in 
accordance  with  the  best  practices3  that  are  fundamental  to  having  a 
reliable  schedule.  In  assessing  each  program’s  cost  estimates,  we  used  the 
GAO  cost  guide  to  evaluate  the  PMOs’  estimating  methodologies, 
assumptions,  and  results  to  determine  whether  the  cost  estimates  were 
comprehensive,  accurate,  well-documented,  and  credible.  We  discussed 
the  results  of  our  assessments  with  the  PMOs,  lead  schedulers,  and  cost 
estimators. 


2GAO- 10-461,  GAO-09-841,  GAO-08-896,  GAO-08-866,  GAO-08-822,  and  GAO-07-860. 

3GAO-08-822  and  GAO-08-896. 

4GAO-09-3SP. 

5GAO-09-3SP. 
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To  address  the  third  objective,  we  obtained  and  reviewed  the  2009  and 
2010  reports6  on  business  transformation  submitted  to  congressional 
defense  committees  by  each  military  service  to  determine  the  extent  to 
which  these  reports  included  performance  measures.  In  addition,  we  met 
with  the  military  departments’  deputy  chief  management  officers  to  obtain 
an  understanding  of  how  they  define  success  in  terms  of  their  respective 
ERPs.  We  also  met  with  the  personnel  within  the  department’s  DCMO 
office  and  the  Director,  Business  Transformation  Agency,  to  obtain  an 
understanding  of  their  respective  roles  and  responsibilities  for  the 
implementation  of  the  ERPs  within  the  department. 

We  conducted  this  performance  audit  from  June  2009  through  October 
2010  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives. 


6United  States  Army,  2009  United  States  Army  Report  to  Congress  (Arlington,  Va.),  and 
2010  Army  Report  to  Congress  on  Business  Transformation  (Arlington,  Va.:  Mar.  1,  2010); 
Department  of  the  Navy,  Congressional  Report,  NDAA  2009,  Section  908,  Business 
Transform  ation  Initiatives  for  the  Military  Departm  ents  (Arlington,  Va.),  and 
Department  of  the  Navy  Fiscal  Year  2010  Business  Transformation  Report  Update 
(Arlington,  Va.);  and  United  States  Air  Force,  Initial  Report  on  Implementation  of  NDAA 
2009,  Business  Transform  ation  Initiatives  for  the  Military  Departments  (Sec  908) 
(Arlington,  Va.:  July  2009),  and  March  2010  Follow-up  Report  on  Implementation  of  NDAA 
2009,  Business  Transformation  Initiatives  for  the  Military  Departments  ( Sec  908) 
(Arlington,  Va.:  March  2010). 
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The  report  number  is  now 
GAO-11-53. 


DEPUTY  CHIEF  MANAGEMENT  OFFICER 
9010  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301-9010 


SEP  2  3  7010 

Mr.  Asif  A.  Khan 

Director, Financial  Management  and  Assurance 
U.S.  Government  Accountability  OfFice 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Mr.  Khan: 

The  Department  of  Defense  (DoD)  response  to  the  U.S.  Government 
Accountability  Office  (GAO)  draft  report  10-951,  “DOD  BUSINESS 
TRANSFORMATION:  Improved  Management  Oversight  of  Business  System 
Modernization  Needed,”  dated  August  25,  2010  (GAO  Code  197086),  is  contained  in  this 
letter.  The  Department  concurs  with  all  GAO  recommendations  contained  in  the  draft 
report. 

Your  audit  highlighted  the  need  to  incorporate  best  practices  into  development  of 
an  integrated  master  schedule  (IMS).  The  Department  recognizes  IMS  as  a  key  program 
management  tool  fundamental  to  program  schedule  reliability.  The  appropriate  Military 
Department  Chief  Management  Officers  will  direct  Program  Managers  to  implement 
these  recommendations  and  have  oversight  over  implementation.  Additionally,  the 
Department  will  issue  guidance  requiring  DoD  business  systems  investments  to  include 
performance  measures  which  can  be  used  to  assess  expected  business  benefits.  For 
strategic  alignment,  these  measures  will  also  be  incorporated  into  the  Business  Enterprise 
Architecture. 


Sincerely, 


Elizabeth  A.  McGrath 
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Tables  6  through  10  provide  information  on  the  status  of  DOD’s  actions  to 
address  our  recommendations  in  previous  reports. 


Table  6:  Status  of  DOD’s  Actions  to  Address  GAO  Recommendations  in  GAO-07-860 


GAO  recommendation 

DOD  action  taken  to  address  the  recommendation 

Status  of  GAO 
recommendation 

1 .  The  Secretary  of  Defense  should  direct  the 
Secretary  of  the  Army  and  the  Director, 

Business  Transformation  Agency  (BTA)  to  jointly 
develop  a  concept  of  operations  that  (1)  clearly 
defines  the  ERP  vision  for  accomplishing  total 
asset  visibility  (TAV)  within  the  Army; 

(2)  addresses  how  its  business  systems  and 
processes,  individually  and  collectively,  will 
provide  the  desired  functionality  to  achieve  total 
asset  visibility;  and  (3)  determines  the  desired 
functionality  among  the  selected  systems. 

The  Army’s  March  201 0  report  to  Congress3  stated  that  the 

Army  lacks  a  concept  of  operations  that  describes  at  a  high 
level  how  the  GFEBS,  GCSS-Army,  and  LMP  systems  relate  to 
each  other  and  how  information  flows  between  and  through  the 
systems.  Furthermore,  the  Army  found  that  representatives  from 
the  three  systems  were  not  able  to  articulate  (1)  what  specific 
data  would  be  exchanged  between  the  three  systems  and 
(2)  which  system  would  be  considered  the  official  system  of 
record  for  master  data  that  needed  to  be  consistent  between  the 
three  systems.  The  Army  did  not  provide  a  timeframe  for 
completing  the  concept  of  operations. 

Open 

2.  The  Secretary  of  Defense  should  direct  the 
Secretary  of  the  Army  and  the  Director,  BTA  to 
jointly  develop  policies,  procedures,  and 
processes  to  support  the  oversight  and 
management  of  selected  groupings  of  business 
systems  that  are  intended  to  provide  a  specific 
capability  or  functionality,  such  as  TAV  from  a 
portfolio  perspective,  utilizing  indicators  such  as 
costs,  schedule,  performance,  and  risks. 

In  June  2010,  the  Under  Secretary  of  the  Army  established  the 
Business  Systems  Information  Technologies  Executive  Steering 
Group.  The  purpose  of  the  group  is  to  advise  the  Army  Chief 
Management  Officer  on  Army-wide  requirements  for  the 
synchronization,  integration,  prioritization,  and  resourcing  of 

Army  business  systems.  The  Army’s  efforts  to  establish  an 
enterprisewide  focus  on  systems  investments  should  improve 
the  Army’s  ability  to  oversee  the  billions  of  dollars  it  is  investing 
in  its  business  systems.  The  group  meets  the  intent  of  the 
recommendation. 

Closed 

3.  The  Secretary  of  Defense  should  direct  the 
Secretary  of  the  Army  and  the  Director,  BTA  to 
jointly  establish  an  independent  verification  and 
validation  (IV&V)  function  for  GFEBS,  GCSS- 
Army,  and  LMP.  Additionally,  direct  that  all  IV&V 
reports  for  each  system  be  provided  to  Army 
management,  the  appropriate  investment  review 
board  (IRB),  and  BTA. 

In  August  2009,  the  Army  awarded  a  contract  to  carry  out  the 
IV&V  function  for  these  systems.  Under  the  contract,  the 
contractor  is  to  provide  reports  on  each  of  the  systems  to  the 
Program  Executive  Office  Enterprise  Information  Systems, 
which  reports  to  the  Army’s  Deputy  Chief  Management  Officer 
(DCMO).  The  Army’s  action  to  establish  an  IV&V  function  under 
the  direction  of  the  Army’s  DCMO,  if  fully  and  effectively 
implemented,  should  enable  the  Army  to  improve  its 
management  and  oversight  of  its  business  systems 
investments. 

Given  the  responsibility  of  the  Chief  Management  Officer  for 
overseeing  and  monitoring  the  implementation  of  Army’s 
business  systems,  the  Army’s  action  meets  the  intent  of  the 
recommendation. 

Closed 

4.  The  Secretary  of  Defense  should  direct  the 
Secretary  of  the  Army  and  the  Director,  BTA  to 
jointly  require  that  any  future  GFEBS  economic 
analysis  identify  costs  and  benefits  in 
accordance  with  the  criteria  specified  by  DOD 
and  Office  of  Management  and  Budget  (OMB) 
guidance,  to  include  a  sensitivity  analysis. 

While  the  Army  has  developed  an  updated  economic  analysis,  it 
was  not  prepared  in  accordance  with  DOD  and  OMB  guidance.13 
For  example,  the  economic  analysis  did  not  include  a  sensitivity 
analysis  or  a  cost  uncertainty  analysis.  Cost  estimates  should 
discuss  any  limitations  of  the  analysis  because  of  uncertainty  or 
biases  surrounding  data  and  assumptions.  Major  assumptions 
should  be  varied,  and  other  outcomes  recomputed  to  determine 
how  sensitive  they  are  to  changes  in  the  assumptions.  Having  a 
range  of  costs  around  a  point  estimate — the  best  guess  at  the 
cost  estimate,  given  the  underlying  data — is  more  useful  to 

Open 
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Status  of  GAO 

GAO  recommendation  DOD  action  taken  to  address  the  recommendation  recommendation 

decision  makers  because  it  conveys  the  level  of  confidence  in 
achieving  the  most  likely  cost  and  also  informs  management  on 
cost,  schedule,  and  risks. 

The  Army  has  stated  that  LMP  system  testers  are  now  Open 

independent  of  the  system  developers.  We  are  in  the  process  of 
evaluating  the  Army’s  actions  as  part  of  our  ongoing  work  on 
the  third  deployment  of  LMP. 


5.  The  Secretary  of  Defense  should  direct  the 
Secretary  of  the  Army  and  the  Director,  BTA  to 
jointly  direct  that  LMP  utilize  system  testers  that 
are  independent  of  the  LMP  system  developers 
to  help  ensure  that  the  system  is  providing  the 
users  of  the  system  the  intended  capabilities. 


Source:  GAO  analysis  of  data  provided  by  DOD. 

“U.S.  Army,  Report  to  Congress  on  Business  Transformation  (Mar.  1 , 2010). 
“Department  of  Defense  Instruction  7041 .3  and  OMB  Circular  No.  A-94. 
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Table  7:  Status  of  DOD’s  Actions  to  Address  GAO  Recommendations  in  GAO-08-822 


GAO  recommendation 

Status  of  GAO 

DOD  action  taken  to  address  the  recommendation  recommendation 

1 .  The  Secretary  of  Defense  direct  the  Secretary 
of  the  Navy  to  ensure  that  investment  in  the  next 
acquisition  phase  of  the  program’s  first  increment 
is  conditional  upon  fully  disclosing  to  program 
oversight  and  approval  entities  the  steps  under 
way  or  planned  to  address  each  of  the  risks 
discussed  in  the  report,  including  the  risk  of  not 
being  architecturally  compliant  and  being 
duplicative  of  related  programs,  not  producing 
expected  mission  benefits  commensurate  with 
reliably  estimated  costs,  not  effectively 
implementing  earned  valued  management  (EVM), 
not  mitigating  known  program  risks,  and  not 
knowing  whether  the  system  is  becoming  more  or 
less  mature  and  stable.  We  further  recommend 
that  investment  in  all  future  Global  Combat 

Support  System-Marine  Corps  (GCSS-MC) 
increments  be  limited  if  the  management  control 
weaknesses  that  are  the  source  of  these  risks, 
and  which  are  discussed  in  the  report,  have  not 
been  fully  addressed. 

An  Enterprise  Risk  Assessment  Methodology  (ERAM)-based  Open 
review  was  conducted  on  GCSS-MC,  and  the  results  were 
presented  at  a  May  2009  IRB  meeting.  According  to  DOD,  the 
assessment  included  a  review  of  the  program’s  risk 
management  database  and  policies.  The  ERAM  process 
identified  seven  risk  areas,  some  of  which  relate  to  risks 
discussed  in  our  report.  DOD  reported  that  the  governance- 
related  risks  identified  in  our  report  require  longer-term 
actions;  while  the  program  had  nevertheless  demonstrated 
compliance  with  its  business  enterprise  architecture  and  that 
the  IRB  reviewed  and  certified  compliance  with  the 
architecture  in  October  2009.  DOD  also  reported  that  the 
program  implemented  a  new  risk  management  process  in 

March  2009  and  developed  metrics  related  to  system  maturity 
and  stability,  such  as  metrics  to  track  defects  during 
developmental  test  and  evaluation,  and  is  tracking  change 
requests  and  generating  monthly  trend  analyses  of  each.  In 
addition,  DOD  reported  that  the  program  is  working  closely 
with  the  Milestone  Decision  Authority,  via  the  IRB,  to  correct 
management  control  weaknesses.  As  of  October  2010,  DOD 
had  yet  to  provide  the  supporting  documentation  for  the  above 
actions  taken  by  the  department. 

2.  The  Secretary  of  Defense  direct  the  appropriate  In  April  201 0,  DOD  reported  that  planning  is  underway  within  Open 
organization  within  DOD  to  collaborate  with  the  BTA  and  the  Office  of  Acquisition  Resources  and  Analysis 


relevant  organizations  to  standardize  the  cost 
element  structure  for  the  department’s  ERP 
programs  and  to  use  this  standard  structure  to 

for  development  of  a  common  set  of  high-level  work  elements, 
such  as  testing,  design,  and  training,  to  augment  detailed  work 
breakdown  structures  developed  by  program  managers  for 

maintain  cost  data  for  its  ERP  programs,  including  their  respective  ERP  programs.  DOD  also  stated  that  it  plans 
GCSS-MC,  and  to  use  this  cost  data  in  developing  to  use  the  common  set  of  high-level  work  elements,  along  with 
future  cost  estimates.  a  common  set  of  cost  elements — buckets  of  cost  types  such 


as  program  management,  technical  labor,  hardware,  and 
software — to  capture  historical  costs  across  ERP  programs. 

DOD  also  stated  that  it  still  plans  to  track  and  maintain  ERP 
cost  data  through  the  Business  Capability  Lifecycle  Integrated 

Management  Information  Environment,  and  use  the  data  to 
develop  future  cost  estimates  and  an  economic  analysis.  As  of 

October  2010,  DOD  did  not  provide  timeframes  for  completion 
of  these  actions. 

3.  The  Secretary  of  Defense  direct  the  Secretary 
of  the  Navy,  through  the  appropriate  chain  of 
command,  to  ensure  that  the  program’s  current 
economic  analysis  is  adjusted  to  reflect  the  risks 
associated  with  it  not  reflecting  cost  data  for 
comparable  ERP  programs,  and  otherwise  not 
having  been  derived  according  to  other  key  cost 
estimating  practices,  and  that  future  updates  to 
the  GCSS-MC  economic  analysis  similarly  do  so. 

In  April  2010,  DOD  reported  that  the  GCSS-MC  program  Open 

developed  its  Cost  Analysis  Requirements  Document  and 

Economic  Analysis  Development  Plan  in  partnership  with  the 

Office  of  the  Secretary  of  Defense  (Cost  Analysis  and 

Program  Evaluation)  to  ensure  that  the  GCSS-MC  economic 
analysis  addresses  DOD-wide  assumptions  and  risks.  DOD 
stated  that  the  independent  cost  estimate  prepared  by  the 

Naval  Center  for  Cost  Analysis  and  approved  in  January  2010 
was  risk  adjusted  and  included  cross-checks  from  similar  ERP 
systems  and  models.  As  of  October  201 0,  DOD  had  yet  to 
provide  the  supporting  documentation  for  the  above  actions. 
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GAO  recommendation 


Status  of  GAO 

DOD  action  taken  to  address  the  recommendation  recommendation 


4.  To  enhance  GCSS-MC’s  use  of  EVM,  we 
recommend  that  the  Secretary  of  Defense  direct 
the  Secretary  of  the  Navy,  through  the  appropriate 
chain  of  command,  to  ensure  that  the  program 
office  (1)  monitors  the  actual  start  and  completion 
dates  of  work  activities  performed  so  that  the 
impact  of  deviations  on  downstream  scheduled 
work  can  be  proactively  addressed;  (2)  allocates 
resources,  such  as  labor  hours  and  material,  to  all 
key  activities  on  the  schedule;  (3)  integrates  key 
activities  and  supporting  tasks  and  subtasks; 

(4)  identifies  and  allocates  the  amount  of  float 
time  needed  for  key  activities  to  account  for 
potential  problems  that  might  occur  along  or  near 
the  schedule’s  critical  path;  (5)  performs  a 
schedule  risk  analysis  to  determine  the  level  of 
confidence  in  meeting  the  program's  activities  and 
completion  date;  (6)  allocates  schedule  reserve 
for  high-risk  activities  on  the  critical  path;  and 
(7)  discloses  the  inherent  risks  and  limitations 
associated  with  any  future  use  of  the  program’s 
EVM  reports  until  the  schedule  has  been  risk- 
adjusted. 


In  April  2010,  DOD  reported  that  the  schedule  was 
rebaselined  and  is  now  used  to  monitor  and  report  actual 
versus  planned  start  and  completion  dates  of  work  activities; 
allocate  resources  to  activities;  integrate  key  activities  and 
supporting  tasks  and  sub-tasks;  identify  and  allocate  the 
amount  of  float  time  needed  for  key  activities,  and  allocate 
schedule  reserve  for  high-risk  activities  on  the  critical  path. 
DOD  also  reported  that  the  program  conducted  a  schedule 
risk  analysis  which  resulted  in  more  detailed  task  definitions, 
the  ability  to  provide  detailed  weekly  status  reports  to  the 
program  manager,  and  more  effective  analysis,  monitoring 
and  risk  assessment  of  the  program’s  scheduled  activities  and 
completion  dates.  As  of  October  2010,  DOD  had  yet  to 
provide  the  supporting  documentation  for  the  above  actions. 


Open 


5.  The  Secretary  of  Defense  direct  the  Secretary 
of  the  Navy,  through  the  appropriate  chain  of 
command,  to  ensure  that  the  program  office 
(1)  adds  each  of  the  risks  discussed  in  this  report 
to  its  active  inventory  of  risks,  (2)  tracks  and 
evaluates  the  implementation  of  mitigation  plans 
for  all  risks,  (3)  discloses  to  appropriate  program 
oversight  and  approval  authorities  whether 
mitigation  plans  have  been  fully  executed  and 
have  produced  the  intended  outcome(s),  and 
(4)  only  closes  a  risk  if  its  mitigation  plan  has  been 
fully  executed  and  produced  the  intended 
outcome(s). 


In  April  2010,  DOD  reported  that  the  program  office  took  a 
number  of  actions  to  strengthen  risk  management.  First,  it 
included  all  risks  reported  by  GAO  as  well  as  risks  identified 
through  DOD’s  ERAM  in  its  risk  database.  Second,  program 
risks  are  now  reviewed,  tracked  and  managed  on  a 
continuous  basis,  and  the  program  office  conducts  weekly  risk 
meetings  to  track  and  evaluate  mitigation  plan  implementation 
for  all  risks.  Third,  monthly  risk  boards  are  convened  to 
discuss  risks  and  mitigation  plans  with  GCSS-MC  senior 
leadership,  and  risks  are  not  closed  without  risk  board 
approval.  Further,  the  program  office  meets  monthly  with  the 
Program  Executive  Office  for  Enterprise  Information  Systems 
and  quarterly  with  the  Assistant  Secretary  of  the  Navy 
(Research  Development  and  Acquisition)  to  discuss  program 
risks.  Also,  the  program  office  reports  risks  to  other  program 
oversight  bodies,  such  as  the  Weapons  Systems  Lifecycle 
Management  and  Materiel  Supply  and  Services  Management 
IRB,  and  the  Defense  Business  Systems  Management 
Committee.  Fourth,  the  program  office  revised  its  Risk 
Management  Plan,  in  March  2009,  to  reflect  these  new 
processes  and  policies.  As  of  October  2010,  DOD  had  yet  to 
provide  the  supporting  documentation  for  the  above  actions. 


Open 
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GAO  recommendation 

DOD  action  taken  to  address  the  recommendation 

Status  of  GAO 
recommendation 

6.  The  Secretary  of  Defense  direct  the  Secretary 

In  April  2010,  DOD  reported  that  during  system  developmental 

Open 

of  the  Navy,  through  the  appropriate  chain  of 
command,  to  ensure  that  the  program  office 

(1)  collects  the  data  needed  to  develop  trends  in 
unresolved  system  defects  and  change  requests 
according  to  their  priority  and  severity  and 

(2)  discloses  these  trends  to  appropriate  program 
oversight  and  approval  authorities. 

test  and  evaluation,  completed  in  October  2009,  the  program 
office  developed  metrics  to  track  defects  and  correction  of 
defects  throughout  the  test  period,  and  that  the  metrics  were 
made  available  to  the  BTA  and  the  Cost  Analysis  and 

Program  Evaluation  Office.  DOD  reported  that  the  program 
office  is  also  (1)  collecting  defect  data  over  time  across 
severity  levels  and  using  diagrams  to  show  trends  and 
(2)  managing  change  requests  according  to  its  configuration 
management  plan  and  generating  trend  analysis  reports  to 
track  them.  As  of  October  201 0,  DOD  had  yet  to  provide  the 
supporting  documentation  for  the  above  actions. 

Source: 

GAO  analysis  of  data  provided  by  DOD. 
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Table  8:  Status  of  DOD’s  Actions  to  Address  GAO  Recommendations  in  GAO-08-866 

GAO  recommendation 

DOD  action  taken  to  address  the  recommendation 

Status  of  GAO 
recommendation 

1 .  The  Secretary  of  Defense  direct  the  Secretary 
of  the  Air  Force  to  direct  Air  Force  program 
management  officials  for  ECSS  and  DEAMS  to 
ensure  that  risk  management  activities  at  all 
levels  of  the  program  are  identified  and 
communicated  to  program  management  to 
facilitate  oversight  and  monitoring.  Key  risks 
described  at  the  appropriate  level  of  detail  should 
include  and  not  be  limited  to  risks  associated  with 
interfaces,  data  conversion,  change 
management,  and  contractor  oversight. 

In  July  2009,  the  DEAMS  Program  Charter  noted  that  risk 
management  activities  at  all  levels  of  the  program  would  be 
identified  and  communicated  to  the  program  manager  to 
facilitate  oversight  and  monitoring.  The  charter  noted  that 
program  risk  will  include,  but  not  be  limited  to,  interfaces,  data 
conversion,  change  management,  and  contractor  oversight. 

The  charter  also  notes  that  the  risk  management  process  will 
include  risk  identified  by  various  reviews,  including  GAO 
audits.  As  of  July  2010,  ECSS  was  still  in  the  process  of 
revising  its  risk  management  plan  to  address  our 
recommendation. 

Open 

2.  The  Secretary  of  Defense  direct  the  Secretary 
of  the  Air  Force  to  direct  the  Air  Force  program 
management  offices  to  test  ECSS  and  DEAMS 
on  relevant  computer  desktop  configurations  prior 
to  deployment  at  a  given  location. 

The  intent  of  the  recommendation  was  to  reduce  program  risk 
and  ensure  that  when  DEAMS  and  ECSS  were  deployed  to  a 
given  location  they  would  operate  as  intended.  According  to 
the  DEAMS  PMO,  the  PMO  has  performed  appropriate  testing 
prior  to  the  system  being  operational  and  if  necessary, 
changes  are  made  prior  to  the  implementation  of  the  system. 
The  Defense  Finance  and  Accounting  Service  is  also 
participating  in  the  testing,  thereby  helping  to  ensure  that  the 
accounting  information  will  process  correctly.  As  of  July  2010, 
ECSS  had  not  yet  become  operational  at  a  given  location. 

Open 

Source:  GAO  analysis  of  data  provided  by  DOD. 
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Table  9:  Status  of  DOD’s  Actions  to  Address  GAO  Recommendations  in  GAO-08-896 


GAO  recommendation 

1 .  The  Secretary  of  Defense  direct  the  Secretary 
of  the  Navy,  through  the  appropriate  chain  of 
command,  to  ensure  that  future  Navy  ERP 
estimates  include  uncertainty  analyses  of 
estimated  benefits,  reflect  the  risks  associated 
with  not  having  cost  data  for  comparable  ERP 
programs,  and  are  otherwise  derived  in  full 
accordance  with  the  other  key  estimating 
practices,  and  economic  analysis  practices 
discussed  in  this  report. 


2.  The  Secretary  of  Defense  direct  the  Secretary 
of  the  Navy,  through  the  appropriate  chain  of 
command,  to  ensure  that  (1)  an  integrated 
baseline  review  on  the  last  two  releases  of  the 
first  increment  is  conducted,  (2)  compliance 
against  the  32  accepted  industry  earned  value 
management  (EVM)  practices  is  verified,  and 
(3)  a  plan  to  have  an  independent  organization 
perform  surveillance  of  the  program’s  EVM 
system  is  developed  and  implemented. 


3.  The  Secretary  of  Defense  direct  the  Secretary 
of  the  Navy,  through  the  appropriate  chain  of 
command,  to  ensure  that  the  schedule 
(1)  includes  the  logical  sequencing  of  all 
activities,  (2)  reflects  whether  all  required 
resources  will  be  available  when  needed, 

(3)  defines  a  critical  path  that  integrates  all  three 
releases,  (4)  allocates  reserve  for  the  high-risk 
activities  on  the  entire  program’s  critical  path,  and 
(5)  incorporates  the  results  of  a  schedule  risk 
analysis  for  all  three  releases  and  recalculates 
program  cost  and  schedule  variances  to  more 
accurately  determine  a  most  likely  cost  and 
schedule  overrun. 


Status  of  GAO 

DOD  action  taken  to  address  the  recommendation  recommendation 

In  July  201 0,  DOD  reported  that  uncertainty  analysis  will  be  Open 
applied  to  the  Navy  ERP’s  benefit  estimate  in  support  of  the 
next  milestone  review,  Full  Deployment  Decision  Review, 
planned  for  the  first  quarter  of  fiscal  year  201 1 .  The  benefit 
estimation  model  is  being  updated  to  include  variations 
among  key  cost  drivers,  such  as  labor  category  efficiency  and 
legacy  system  sustainment  difficulty  factors,  through  the  use 
of  Monte  Carlo  simulation.  In  addition,  DOD  reported  that  the 
Navy  ERP  program  is  working  with  the  Space  and  Naval 
Warfare  Systems  Command  and  Naval  Center  for  Cost 
Analysis,  as  they  conduct  an  independent  assessment  of  the 
program’s  life-cycle  cost  estimate.  According  to  DOD,  the 
assessment  will  include  a  review  of  the  risk/uncertainty 
approach  and  methodologies  used  to  develop  the  cost 
estimate. 

In  July  2010,  DOD  reported  that  the  Navy  ERP  program  office  Open 
conducted  an  integrated  baseline  review  of  its  second 
release,  which  resulted  in  recommendations  to  mature  and 
implement  EVM  processes.  Because  the  third  release  is  no 
longer  a  part  of  Navy  ERP’s  program  of  record,  this 
recommendation  is  not  applicable  to  this  release.  In  addition, 

DOD  reported  that  the  Navy  Center  for  Earned  Value 
Management  planned  to  conduct  surveillance  of  the  Navy 
ERP’s  EVM  system  in  September  201 0,  and  that  it  would 
review  compliance  against  the  32  accepted  industry  EVM 
practices. 

As  of  July  201 0,  Navy  ERP  continues  to  make  progress  in  Open 
addressing  this  recommendation.  For  example,  it  is  using 
metrics  to  track  and  logically  link  activities  and  account  for 
resources  and  their  availability,  and  it  plans  to  conduct  a 
schedule  risk  assessment  in  September  2010  so  that 
reserves  can  be  established  for  high-risk  activities.  Further,  in 
July  2010,  DOD  reported  that  it  was  not  feasible  to  define  a 
critical  path  integrating  all  three  releases  because  (1)  key 
functionality  deliverables  for  the  first  release  were  completed 
prior  to  the  second  release’s  development  and  (2)  the  third 
release  was  removed  from  Navy  ERP’s  program  of  record. 

However,  the  March  2010  metrics  report  shows  that  not  all 
activities  are  logically  sequenced,  which  can  affect  the 
calculation  of  the  critical  path  and  finish  date.  Further, 
because  the  schedules  are  not  integrated  and  personnel  are 
assigned  to  activities  across  multiple  releases,  if  deployment 
activities  in  one  schedule  were  to  be  delayed,  the  other 
schedule  that  requires  the  same  resources  would  likely  also 
be  delayed. 
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Status  of  GAO 


GAO  recommendation 


DOD  action  taken  to  address  the  recommendation  recommendation 


4.  The  Secretary  of  Defense  direct  the  Secretary  The  department  has  taken  actions  to  address  the  intent  of  Closed 

of  the  Navy,  through  the  appropriate  chain  of  this  recommendation.  First,  the  Navy  ERP  program  office 

command,  to  ensure  that  (1)  the  plans  for  reevaluated  its  plans  for  mitigating  risks  associated  with  data 

mitigating  the  risks  associated  with  converting  conversion  and  adopting  new  business  processes.  Second, 

data  from  legacy  systems  to  Navy  ERP  and  the  program  manager  and  System  Command  officials  report 

positioning  the  commands  for  adopting  the  new  monthly  to  the  Navy  ERP  Senior  Integration  Board  (NESIB) 

business  processes  embedded  in  the  Navy  ERP  on  performance,  and  periodically  brief  oversight  and  approval 

are  re-evaluated  in  light  of  the  recent  experience  authorities  on  the  implementation  of  risk  mitigation  plans. 

with  the  Naval  Air  Systems  Command  (NAVAIR)  Third,  the  NESIB  requires  actionable  reporting  on 

and  adjusted  accordingly,  (2)  the  status  and  performance  by  the  program  manager  and  System  Command 

results  of  these  and  other  mitigation  plans’  officials,  and  the  program  manager  is  to  report  to  the 

implementation  are  periodically  reported  to  Milestone  Decision  Authority  on  implementation  of  risk 

program  oversight  and  approval  authorities,  mitigation  strategies.  Fourth,  the  program's  risk  inventory  has 

(3)  these  authorities  ensure  that  those  entities  been  updated  to  include  risks  related  to  adopting  new 

responsible  for  implementing  these  strategies  are  business  processes  and  data  conversion. 

held  accountable  for  doing  so,  and  (4)  each  of  the 

risks  discussed  in  this  report  are  included  in  the 

program’s  inventory  of  active  risks  and  managed 


accordingly. 


Source:  GAO  analysis  of  data  provided  by  DOD. 


Page  67 


GAO-11-53  DOD  Business  Systems 


Appendix  III:  Status  of  DOD’s  Actions  on 

Previous  GAO  Recommendations  Related  to 

Business  Systems  Modernization 

Table  10:  Status  of  DOD’s  Actions  to  Address  GAO  Recommendations  in  GAO-09-841 

GAO  recommendation 

DOD  action  taken  to  address  the  recommendation 

Status  of  GAO 
recommendation 

1 .  The  Secretary  of  Defense  direct  the  Secretary 
of  the  Navy,  through  the  appropriate  chain  of 
command,  to  (1)  revise  the  Navy  ERP 
procedures  for  controlling  system  changes  to 
explicitly  require  that  a  proposed  change’s  life- 
cycle  cost  impact  be  estimated  and  considered  in 
making  change  request  decisions  and  (2)  capture 
the  cost  and  schedule  impact  of  each  proposed 
change  in  the  Navy  ERP  automated  control 
tracking  tool. 

The  Navy  ERP  program  updated  its  Enterprise  Change 

Request  Process  and  Procedures  to  explicitly  require  that  a 
change’s  life-cycle  cost  impact  be  estimated  as  part  of  the 
change  control  process.  In  addition,  the  change  control 
tracking  tool  now  captures  cost  and  schedule  impact 
information.  As  a  result,  management  of  the  Navy  ERP’s 
change  control  process  has  been  strengthened.  As  a  result, 
approval  authorities  should  be  provided  key  information 
needed  to  fully  inform  their  decisions  on  whether  to  approve  a 
change,  thus  decreasing  the  risk  of  unwarranted  cost 
increases  and  schedule  delays. 

Closed 

2.  The  Secretary  of  Defense  direct  the  Secretary 
of  the  Navy,  through  the  appropriate  chain  of 
command,  to  (1)  stop  performance  of  the  IV&V 
function  under  the  existing  contract  and 
(2)  engage  the  services  of  an  IV&V  agent  that  is 
independent  of  all  Navy  ERP  management, 
development,  testing,  and  deployment  activities 
that  it  may  review. 

According  to  DOD,  the  Navy  ERP  program  office  terminated 
the  IV&V  functions  under  the  existing  contract  on 

September  30,  2009,  and  awarded  a  new  IV&V  contract  in 
September  2010. 

Closed 

Source:  GAO  analysis  of  data  provided  by  DOD. 
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This  appendix  provides  the  results  of  our  analysis  of  the  extent  to  which 
the  processes  and  methodologies  used  to  develop  and  maintain  the  four 
ERP  integrated  master  schedules  meet  the  nine  best  practices  associated 
with  effective  schedule  estimating.1  Tables  11,  12,  13,  14,  and  15  provide 
the  detailed  results  of  our  analyses  of  the  program  schedules  for  DEAMS, 
ECSS,  GFEBS,  and  GCSS-Army  compared  to  the  nine  best  practices. 

“Not  met”  means  the  program  provided  no  evidence  that  satisfies  any  of 
the  criterion.  “Minimally”  means  the  program  provided  evidence  that 
satisfies  a  small  portion  of  the  criterion.  “Partially”  means  the  program 
provided  evidence  that  satisfies  about  half  of  the  criterion.  “Substantially” 
means  the  program  provided  evidence  that  satisfies  a  large  portion  of  the 
criterion.  “Fully  met”  means  the  program  provided  evidence  that 
completely  satisfies  the  criterion. 


Table  1 1 :  Analysis  of  the  Air  Force’s  DEAMS  Program  Schedule 


Best  practice  Explanation 


Criterion  met  GAO  analysis 


1 .  Capturing  all  The  schedule  should  reflect  all  Partially 
activities  activities  as  defined  in  the 

project’s  work  breakdown 
structure,  which  defines  in 
detail  the  work  necessary  to 
accomplish  a  project’s 
objectives,  including  activities 
to  be  performed  by  both  the 
owner  and  contractors. 


Our  analysis  found  that  the  DEAMS  program  schedule  is  not 
fully  integrated.  While  the  DEAMS  PMO  maintains  internal 
schedules  that  reflect  government-only  activities,  these 
government  schedules  have  no  links  to  activities  within  the 
contractor  schedule.  We  found  that  activities  in  the  contractor 
schedule  are  mapped  to  contract  line  item  numbers  and 
assigned  to  integrated  product  teams,  but  many  activities  are 
missing  contractor  work  breakdown  structure  mappings. 

PMO  officials  told  us  that  because  of  the  firm-fixed  price  (FFP) 
nature  of  the  current  contract,  the  prime  contractor  is  not 
obligated  to  provide  detailed  insight  into  the  contractor  schedule. 
Instead,  the  PMO  uses  the  contractor  schedule  as  a  starting 
point  to  develop  more  detailed  internal  tools,  such  as  lower-level 
schedule  information  maintained  in  spreadsheets.  But  without 
government  activities  fully  integrated  with  contractor  activities, 
we  cannot  guarantee  that  the  schedule  has  either  adequately 
captured  all  key  activities  necessary  for  the  program’s 
completion  or  that  the  PMO  can  reliably  estimate  the  finish  date 
for  the  program. 


‘GAO-Og-SSP. 
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Best  practice  Explanation 


Criterion  met  GAO  analysis 


2.  Sequencing  The  schedule  should  be  Minimally 

all  activities  planned  so  that  critical  project 
dates  can  be  met.  To  meet  this 
objective,  activities  need  to  be 
logically  sequenced — that  is, 
listed  in  the  order  in  which  they 
are  to  be  carried  out.  In 
particular,  activities  that  must 
be  completed  before  other 
activities  can  begin 
(predecessor  activities),  as  well 
as  activities  that  cannot  begin 
until  other  activities  are 
completed  (successor 
activities),  should  be  identified. 

This  helps  ensure  that 
interdependencies  among 
activities  that  collectively  lead 
to  the  accomplishment  of 
events  or  milestones  can  be 
established  and  used  as  a 
basis  for  guiding  work  and 
measuring  progress. 


Our  analysis  of  the  DEAMS  contractor  schedule  shows  that  131 
of  the  273  remaining  activities,  or  48  percent,  have  missing 
predecessor  or  successor  logic.  Missing  predecessors  or 
successors  reduce  the  credibility  of  the  calculated  dates.  If  an 
activity  that  has  no  logical  successor  slips,  the  schedule  will  not 
reflect  the  effect  on  the  critical  path,  float,  or  scheduled  start 
dates  of  downstream  activities.  In  addition,  we  found  that  42 
remaining  activities,  or  15  percent,  have  “dangling”  logic — that  is, 
these  activities  whose  start  or  finish  dates  are  missing  logic.  Of 
these  42  activities  with  dangling  logic,  37  activities  are  missing 
logic  that  would  determine  their  start  dates.  Because  their  start 
dates  are  not  determined  by  logic,  these  activities  would  have  to 
start  earlier  in  order  to  finish  on  time  if  they  ran  longer  than  their 
planned  durations.  The  other  5  activities  with  dangling  logic  are 
missing  successors  off  their  finish  date.  In  other  words,  these 
activities  could  continue  indefinitely  and  not  affect  the  start  or 
finish  dates  of  future  activities. 

We  found  six  remaining  activities  with  start-to-finish  links. 
Start-to-finish  links  are  rarely,  if  ever,  used  because  they  have 
the  odd  effect  of  causing  a  successor  to  finish  before  its 
predecessor.3  We  also  found  18  links  to  or  from  summary  tasks. 
Summary  tasks  should  not  have  dependencies  because  they 
take  their  start  date,  finish  date,  and  duration  from  lower-level 
activities. 

In  addition,  we  found  50  remaining  activities  (18  percent)  with 
Start  No  Earlier  Than  constraints.  These  are  considered  “soft" 
date  constraints  in  that  they  allow  the  activity  to  slip  into  the 
future  based  on  what  happens  to  their  predecessor  activities. 
While  activities  may  be  soft  constrained,  for  example,  to 
represent  receipt  of  delivery  of  equipment,  in  general 
constraining  an  activity’s  start  date  prevents  managers  from 
accomplishing  work  as  soon  as  possible  and  consumes  flexibility 
in  the  project. 

Of  the  remaining  activities,  47  activities  are  linked  to  their 
successor  activities  with  lags,  including  lags  that  are  greater 
than  100  days.  Lags  represent  the  passing  of  time  between 
activities  but  are  often  misused  to  put  activities  on  a  specific  date 
or  to  insert  a  buffer  for  risk.  Lags  should  be  justified  because 
they  cannot  vary  with  risk  or  uncertainty. 

PMO  officials  noted  that  the  contractor  schedule  follows  a  Data 
Item  Description  (DID)  that  details  the  preparation  of  the 
schedule.  However,  the  schedule  does  not  meet  the 
requirements  set  forth  in  the  DID.  For  example,  the  DID  states 
that  the  schedule  “shall  be  an  integrated,  logical  network-based 
schedule”  and  that  a  key  element  of  the  schedule  is  the 
“relationship/dependency”  of  an  activity.  Without  logically 
sequencing  activities,  the  schedule  cannot  be  used  as  a  reliable 
basis  for  guiding  work  and  measuring  progress. 
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Best  practice 

Explanation 

Criterion  met 

GAO  analysis 

3.  Assigning 
resources  to  all 
activities 

The  schedule  should  reflect 
what  resources  (e.g.,  labor, 
materials,  and  overhead)  are 
needed  to  do  the  work, 
whether  all  required  resources 
will  be  available  when  needed, 
and  whether  any  funding  or 
time  constraints  exist. 

Fully  met 

Because  of  the  current  FFP  contractual  arrangement,  the 
government  does  not  have  insight  into  the  contractor's  efforts  to 
assign  resources  to  activities.  However,  contractor  officials 
provided  evidence  that  resources  have  been  assigned  to 
activities  within  their  schedule.  In  addition,  PMO  officials  assign 
and  monitor  individual  government  resources  to  lower-level 
activities  that  are  updated  in  internal  tools  outside  the  delivered 
contractor  schedule. 

4.  Establishing  The  schedule  should  Substantially  The  majority  of  remaining  activities  in  the  contractor  schedule 

the  duration  of  realistically  reflect  how  long  meet  best  practices  for  durations.  There  are  50  activities  (18 

all  activities  each  activity  will  take  to  percent)  with  planned  durations  longer  than  44  days,  which 

execute.  In  determining  the  exceeds  the  best  practice  for  activity  duration.b  There  are  7 

duration  of  each  activity,  the  (3  percent)  level-of-effort  activities  with  durations  greater  than 

same  rationale,  historical  data,  1 ,200  days.  These  level-of-effort  activities  drive  the  end  date  of 

and  assumptions  used  for  cost  the  project  and  hence  adversely  affect  the  calculation  of  the 

estimating  should  be  used.  critical  path — the  longest  duration  path  through  the  sequenced 

Durations  should  be  as  short  list  of  activities.  Level-of-effort  activities,  such  as  systems 

as  possible  and  have  specific  engineering  and  program  management,  should  not  define  the 

start  and  end  dates.  The  critical  path  because  they  are  nondiscrete  support  activities  that 

schedule  should  be  continually  do  not  produce  a  definite  end  product, 

monitored  to  determine  when 
forecasted  completion  dates 
differ  from  planned  dates;  this 
information  can  be  used  to 
determine  whether  schedule 
variances  will  affect 
subsequent  work. 
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Best  practice  Explanation 


Criterion  met  GAO  analysis 


5.  Integrating 
schedule 
activities 
horizontally  and 
vertically 


The  schedule  should  be 
horizontally  integrated, 
meaning  that  it  should  link 
products  and  outcomes 
associated  with  other 
sequenced  activities.  These 
links  are  commonly  referred  to 
as  “handoffs”  and  serve  to 
verify  that  activities  are 
arranged  in  the  right  order  to 
achieve  aggregated  products 
or  outcomes.  The  schedule 
should  also  be  vertically 
integrated,  meaning  that  the 
dates  for  starting  and 
completing  activities  in  the 
integrated  master  schedule 
should  be  aligned  with  the 
dates  for  supporting  tasks  and 
subtasks.  Such  mapping  or 
alignment  among  levels 
enables  different  groups  to 
work  to  the  same  master 
schedule. 


Minimally 


Vertical  integration — that  is,  the  ability  to  consistently  trace  work 
breakdown  structure  elements  between  detailed,  intermediate, 
and  master  schedules — is  demonstrated  somewhat  because  of 
the  efforts  by  the  DEAMS  PMO  to  enhance  its  insight  into 
contractor  effort  despite  the  FFP  contract  environment.  PMO 
officials  stated  that  while  the  contractor  is  under  no  obligation  to 
provide  detailed  activities  in  the  contractor  schedule,  the 
government  has  broken  down  areas  such  as  object  development 
and  testing  into  detailed  activities  with  internal  tools  that  allow  for 
weekly  monitoring  and  status  checking.  However,  we  could  not 
fully  establish  the  link  between  the  internal  updating  of  activities 
by  the  government  in  lower-level  spreadsheets  and  the  high- 
level  schedule  delivered  by  the  contractor. 

Issues  with  missing  dependencies,  activities  with  dangling  logic, 
overuse  of  lags,  and  critical  level-of-effort  activities  prevent  the 
contractor  schedule  from  fully  complying  with  the  requirement  of 
horizontal  integration — that  is,  the  overall  ability  of  the  schedule 
to  depict  relationships  between  different  program  elements  and 
product  handoffs.  PMO  officials  stated  that  rather  than  using  the 
high-level  contractor  schedule,  government  and  contractor 
subject  matter  experts  meet  each  week  to  discuss  progress  on 
ongoing  activities  using  other  internal  management  tools.  If 
activities  are  delayed  or  accelerated,  the  experts  discuss 
potential  impacts  to  downstream  activities  and  provide 
management  with  weekly  to  daily  information  on  these  impacts. 
But  while  subject  matter  experts  may  understand  the  impacts  of 
delayed  activities,  senior  decision  makers  may  not  be  aware  of 
near-critical  activities  that  have  the  potential  to  significantly  delay 
the  project,  nor  do  they  have  the  proper  insight  into  available 
float — the  amount  of  time  an  activity  can  slip  before  it  delays  the 
finish  date  of  the  project — that  can  be  used  to  mitigate  the  risk  of 
critical  or  near-critical  activities. 
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Best  practice  Explanation  Criterion  met 

6.  Establishing  Scheduling  software  should  be  Minimally 
the  critical  path  used  to  identify  the  critical 
for  all  activities  path,  which  represents  the 
chain  of  dependent  activities 
with  the  longest  total  duration. 

Establishing  a  project’s  critical 
path  is  necessary  to  examine 
the  effects  of  any  activity 
slipping  along  this  path. 

Potential  problems  along  or 
near  the  critical  path  should 
also  be  identified  and  reflected 
in  scheduling  the  duration  of 
high-risk  activities. 


7.  Identifying  The  schedule  should  identify  Minimally 
reasonable  float  the  float — the  amount  of  time 
by  which  a  predecessor  activity 
can  slip  before  the  delay 
affects  successor  activities — so 
that  a  schedule’s  flexibility  can 
be  determined.  As  a  general 
rule,  activities  along  the  critical 
path  have  the  least  float.  Total 
float  is  the  total  amount  of  time 
by  which  an  activity  can  be 
delayed  without  delaying  the 
project’s  completion,  if 
everything  else  goes  according 
to  plan. 


GAO  analysis 

Our  analysis  could  not  determine  a  valid  critical  path  within  the 
DEAMS  contractor  schedule,  particularly  because  over  60 
percent  of  remaining  activities  have  missing  or  incomplete  logic, 
and  because  level-of-effort  activities  (over  1 ,200  days  long) 
define  the  start  and  finish  dates  of  the  project.  Level-of-effort 
activities,  such  as  systems  engineering  and  program 
management,  should  not  define  the  critical  path  because  they 
are  nondiscrete  support  activities  that  do  not  produce  a  definite 
end  product. 

PMO  officials  acknowledged  that  a  critical  path  cannot  be 
calculated  within  the  schedule  and  stated  that  the  contractor 
schedule  is  used  only  as  a  starting  point  for  more  detailed 
internal  tracking  tools  such  as  spreadsheets.  Detail  is  not 
available  within  the  contractor  schedule  because  of  the  current 
FFP  contract  with  the  contractor.  PMO  officials  also  stated  that 
establishing  a  traditional  critical  path  is  not  possible  in  a  complex 
ERP  environment  because  there  is  no  one  clear  path  through 
development  or  testing.  Rather  than  use  the  high-level 
contractor  schedule  government  and  contractor  subject  matter 
experts  meet  on  a  weekly  to  daily  basis  to  discuss  progress  on 
ongoing  activities  using  other  internal  management  tools.  If 
activities  are  delayed  or  accelerated,  the  experts  discuss 
potential  impacts  to  downstream  activities  and  provide 
management  with  weekly  to  daily  information  on  these  impacts. 
But  senior  decision  makers  may  not  be  aware  of  near-critical 
activities  nor  have  the  proper  insight  into  available  float  that  can 
be  used  to  mitigate  the  risks  associated  with  these  activities.  In 
addition,  PMO  officials  noted  that  the  contractor  schedule  follows 
a  DID  that  details  the  preparation  of  the  schedule.  However,  the 
contractor  schedule  does  not  meet  the  requirements  set  forth  in 
the  DID.  The  DID  states  a  critical  path  is  a  key  element  of  the 
detailed  schedule;  that  “the  critical  path  and  near-critical  paths 
are  calculated  by  the  scheduling  software”;  and  “the  critical  path 
shall  be  easily  identified.” 

Our  analysis  found  that  float  calculations  within  the  DEAMS 
contractor  schedule  are  not  reliable  because  of  the  improper 
linking  of  summary  tasks.  In  addition,  because  the  schedule  is 
missing  dependencies,  float  estimates  will  be  miscalculated 
because  float  is  directly  related  to  the  logical  sequencing  of 
events.  PMO  officials  told  us  that  internal  activity  tracking  and 
monitoring  tools  used  in  lieu  of  the  detailed  contractor  activities 
do  not  allow  insight  into  float  calculations. 

PMO  officials  noted  that  the  contractor  schedule  follows  a  DID 
that  details  the  preparation  of  the  schedule.  However,  the 
contractor  schedule  does  not  meet  the  requirements  set  forth  in 
the  DID.  The  DID  states  total  float  is  a  key  element  of  the 
detailed  schedule  to  be  delivered  monthly.  Without  float 
estimates  management  may  be  unable  to  allocate  resources 
from  non-critical  activities  to  activities  that  cannot  slip  without 
affecting  the  project  finish  date. 
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Best  practice  Explanation 


Criterion  met  GAO  analysis 


8.  Conducting  a  A  schedule  risk  analysis  should  Minimally 
schedule  risk  be  performed  using  statistical 
analysis  techniques  to  predict  the  level 

of  confidence  in  meeting  a 
project’s  completion  date.  This 
analysis  focuses  not  only  on 
critical  path  activities  but  also 
on  activities  near  the  critical 
path,  since  they  can  affect  the 
project’s  status. 


The  program  office  has  not  performed  a  schedule  risk  analysis 
on  the  schedule  because  the  schedule  is  not  used  as  a  primary 
tool  for  monitoring  the  status  of  the  program.  However,  program 
officials  stated  that  they  have  tied  risks  to  what  subject  matter 
experts  consider  to  be  critical  path  activities.  They  stated  that 
they  proactively  monitor  risk  on  a  weekly  basis  by  assigning  a 
probability  to  the  risk,  examining  the  potential  impact  of  the  risk 
on  activities  if  it  is  realized,  and  developing  mitigation  plans  to  be 
executed  if  the  risk  is  realized.  However,  the  risk  assessments 
cannot  be  used  to  calculate  the  overall  probability  of  finishing  the 
project  on  time.  Since  any  task  can  become  critical  if  it  is 
delayed  long  enough,  complete  schedule  logic  and  a 
comprehensive  risk  assessment  are  essential  tools  for  decision 
makers.  A  schedule  risk  analysis  can  be  used  to  determine  a 
level  of  confidence  in  meeting  the  completion  date  or  whether 
proper  reserves  have  been  incorporated  into  the  schedule.  A 
schedule  risk  analysis  will  calculate  schedule  reserve,  which  can 
be  set  aside  for  those  activities  identified  as  high  risk.  Without 
this  reserve,  the  program  faces  the  risk  of  delays  to  the 
scheduled  completion  date  if  any  delays  were  to  occur  on  critical 
path  activities. 

In  addition,  PMO  officials  noted  that  the  contractor  schedule 
follows  a  DID  that  details  the  preparation  of  the  schedule. 
However,  the  contractor  schedule  does  not  meet  the 
requirements  set  forth  in  the  DID.  The  DID  states  that  a  key 
element  of  the  detailed  schedule  is  a  schedule  risk  analysis  that 
“predicts  the  probability  of  project  completion  by  contractual 
dates”  using  three-point  estimates  about  the  remaining  durations 
of  remaining  activities. 
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Best  practice  Explanation 


Criterion  met  GAO  analysis 


9.  Updating  the 
schedule  using 
logic  and 
durations  to 
determine  the 
dates 


The  schedule  should  be  Minimally 

continuously  updated  using 
logic  and  durations  to 
determine  realistic  start  and 
completion  dates  for  program 
activities.  The  schedule  should 
be  analyzed  continuously  for 
variances  to  determine  when 
forecasted  completion  dates 
differ  from  planned  dates.  This 
analysis  is  especially  important 
for  those  variations  that  impact 
activities  identified  as  being  in 
a  project's  critical  path  and  can 
impact  a  scheduled  completion 
date. 


Our  analysis  shows  the  contractor  schedule  does  not  have  a 
status  date  (or  data  date),  nor  did  the  program  office  expect  one. 
A  status  date  denotes  the  date  of  the  latest  update  to  the 
schedule  and  thus  defines  the  point  in  time  at  which  completed 
work  and  remaining  work  are  calculated.  Officials  stated  that  the 
status  date  is  reflected  by  the  month  in  the  schedule  file  name; 
but  because  no  day  is  given  there  is  no  indication  whether  the 
date  reflects  the  beginning  or  end  of  the  calendar  month  or 
beginning  or  end  of  the  contractor  accounting  period. 

Regardless  of  the  exact  date,  we  found  31  activities  that  had 
actual  starts  in  future  months  relative  to  the  month  in  the  file 
name.  That  is,  according  to  the  schedule,  these  activities  had 
actually  started  in  the  future.  For  example,  the  schedule  file 
name  is  November  2009,  yet  we  found  actual  start  dates  for 
activities  in  December  2009,  February  201 0,  and  April  201 0. 

PMO  officials  noted  that  the  contractor  schedule  follows  the  DID 
that  details  the  preparation  of  the  schedule.  Flowever,  the 
contractor  schedule  does  not  meet  the  requirements  set  forth  in 
the  DID.  The  DID  states  that  “actual  start  and  actual  finish  dates, 
as  recorded,  shall  not  be  later  than  the  status  date.” 

PMO  officials  stated  that  rather  than  use  the  high-level 
contractor  schedule,  which  does  not  give  the  required  activity 
detail,  government  and  contractor  subject  matter  experts  meet 
on  a  weekly  to  daily  basis  to  discuss  progress  on  ongoing 
activities  using  other  internal  management  tools.  For  example, 
the  Testing  Integrated  Product  Team  meets  daily  to  review  tasks 
that  have  been  performed  that  day.  If  deadline  criteria  are  not 
met,  senior  decision  makers  are  alerted  to  potential  impacts  to 
the  schedule.  Flowever,  the  schedule  should  use  logic  and 
durations  in  order  to  reflect  realistic  start  and  completion  dates 
for  program  activities.  The  schedule  should  also  be  continually 
monitored  to  determine  when  forecasted  completion  dates  differ 
from  the  planned  dates,  which  can  be  used  to  determine 
whether  schedule  variances  will  affect  downstream  work. 
Maintaining  the  integrity  of  the  schedule  logic  is  not  only 
necessary  to  reflect  true  status,  but  is  also  required  before 
conducting  a  schedule  risk  analysis. 


Source:  GAO  analysis  based  on  data  provided  by  the  DEAMS  PMO. 

“Activities  need  to  have  certain  predecessor-successor  relationships  so  the  schedule  gives  the  correct 
results  when  they  are  updated  or  when  durations  change.  Two  logic  requirements  have  to  be 
provided:  (1)  a  finish-to-start  or  start-to-start  predecessors,  so  that  if  the  activity  is  longer  than 
scheduled  it  does  not  just  start  earlier  automatically,  and  (2)  finish-to-start  or  finish-to-finish 
successors  that  will  be  “pushed”  if  they  take  longer  or  finish  later. 

“The  Naval  Air  Systems  Command  recommends  keeping  individual  task  durations  to  less  than  two 
calendar  months  (44  working  days).  The  shorter  the  duration  of  the  tasks  in  the  schedule,  the  more 
often  the  Control  Account  Managers  are  compelled  to  update  completed  work  which  more  accurately 
reflects  the  actual  status  of  the  tasks.  When  task  durations  are  too  long,  management  insight  into  the 
actual  status  of  the  activity  is  reduced. 
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Table  12:  Analysis  of  the  Air  Force’s  ECSS  Solutions  Development  Project  Schedule 


Best  practice 

Explanation 

Criterion  met 

GAO  analysis 

1 .  Capturing  all 
activities 

The  schedule  should  reflect  all 
activities  as  defined  in  the 
project’s  work  breakdown 
structure,  which  defines  in 
detail  the  work  necessary  to 
accomplish  a  project’s 
objectives,  including  activities 
to  be  performed  by  both  the 
owner  and  contractors. 

Substantially 

While  the  PMO  does  have  detailed  schedules  of  government 
effort — a  commendable  best  practice — these  are  not  fully 
integrated  into  an  integrated  master  schedule  (IMS)  with  the 
contractor  schedules.  Our  analysis  found  that  the  ECSS 

Solutions  Development  schedule  contains  215  detail  activities 
associated  with  government  effort,  representing  dependencies 
between  contractor  and  government  activities.  However,  the 
government  activities  are  not  completely  linked  to  government 
schedules  maintained  and  updated  by  the  government  PMO. 

Our  analysis  found  that  activities  in  the  Solutions  Development 
workstream  schedule  are  mapped  to  contractor  work  breakdown 
structure  elements  and  can  be  traced  to  completion  criteria  and 
descriptions  of  associated  work  products. 

Page  76 


GAO-11-53  DOD  Business  Systems 


Appendix  IV:  Assessments  of  Four  DOD  ERP 
Programs’  Integrated  Master  Schedules 


Best  practice  Explanation 


Criterion  met  GAO  analysis 


2.  Sequencing  The  schedule  should  be  Partially 

all  activities  planned  so  that  critical  project 
dates  can  be  met.  To  meet  this 
objective,  activities  need  to  be 
logically  sequenced — that  is, 
listed  in  the  order  in  which  they 
are  to  be  carried  out.  In 
particular,  activities  that  must 
be  completed  before  other 
activities  can  begin 
(predecessor  activities),  as  well 
as  activities  that  cannot  begin 
until  other  activities  are 
completed  (successor 
activities),  should  be  identified. 

This  helps  ensure  that 
interdependencies  among 
activities  that  collectively  lead 
to  the  accomplishment  of 
events  or  milestones  can  be 
established  and  used  as  a 
basis  for  guiding  work  and 
measuring  progress. 


Our  analysis  shows  that  31  of  the  1 ,901  remaining  activities,  or  2 
percent,  have  missing  predecessor  or  successor  logic.  This  is  a 
relatively  low  number  for  such  a  highly  integrated  schedule,  but 
any  number  of  missing  predecessors  or  successors  can  reduce 
the  credibility  of  calculated  dates.  If  an  activity  that  has  no  logical 
successor  slips,  the  schedule  will  not  reflect  the  effect  on  the 
critical  path,  float,  or  scheduled  start  dates  of  future  activities. 
However,  of  those  remaining  activities  that  have  logical 
predecessor  and  successor  links,  259  activities  (14  percent), 
have  “dangling  logic.”  Of  these  259  activities  with  dangling  logic, 
229  activities  are  missing  logic  that  would  determine  their  start 
dates.  Because  their  start  dates  are  not  determined  by  logic, 
these  activities  would  have  to  start  earlier  in  order  to  finish  on 
time  if  they  ran  longer  than  their  planned  durations.  The  other  30 
activities  with  dangling  logic  are  missing  successors  off  their 
finish  date.  In  other  words,  these  activities  could  continue 
indefinitely  and  not  affect  the  start  or  finish  dates  of  downstream 
activities. 

The  schedule  includes  four  Must  Finish  On  constraints.  A  Must 
Finish  On  constraint  is  considered  a  “hard”  date  constraint 
because  it  prevents  the  activity  from  finishing  earlier  or  later  than 
its  planned  date.  This  renders  the  schedule  rigid  and  prevents 
the  schedule  from  being  dynamic.  A  Must  Finish  On  constraint  is 
artificial  and  makes  the  scheduled  activity  appear  to  be  on  track 
to  finish  on  time  when  it  may  not  be.  There  are  also  1 7  Start  No 
Earlier  Than  constraints  within  the  schedule.  These  are 
considered  “soft”  constraints  in  that  they  allow  the  activity  to  slip 
into  the  future  based  on  what  happens  to  their  predecessor 
activities.  Activities  may  be  soft  constrained,  for  example,  to 
represent  receipt  of  delivery  of  equipment.  However,  in  general 
constraining  an  activity’s  start  date  prevents  managers  from 
accomplishing  work  as  soon  as  possible  and  consumes  flexibility 
in  the  project. 

We  found  78  start-to-finish  links  within  the  schedule.  Start-to- 
finish  links  are  rarely,  if  ever,  used  in  scheduling  practice 
because  they  have  the  odd  effect  of  causing  a  successor  activity 
to  finish  when  its  predecessor  starts.  Moreover,  we  found  that 
each  of  the  78  start-to-finish  links  have  3-to  7-day  lags.  That  is, 
the  schedule  logic  dictates  that  these  successors  must  finish  a 
set  number  of  days  before  their  predecessors  begin.  Start-to- 
finish  logic  is  at  best  confusing  and  at  worst  incorrect;  activities 
should  be  rearranged  to  find  true  predecessors  and  successors 
and  linked  with  straightforward  logic. 

Of  the  1 ,901  remaining  activities,  467  activities  are  linked  to  their 
successor  activities  with  lags.  Lags  represent  the  passing  of  time 
between  activities  but  are  often  misused  to  put  activities  on  a 
specific  date  or  to  insert  a  buffer  for  risk.  Lags  should  be  justified 
because  they  cannot  have  risk  or  uncertainty.  Without  logically 
sequencing  activities,  the  schedule  cannot  be  used  as  a  reliable 
basis  for  guiding  work  and  measuring  progress. 
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Best  practice  Explanation  Criterion  met 

3.  Assigning  The  schedule  should  reflect  Minimally 

resources  to  all  what  resources  (e.g.,  labor, 
activities  materials,  and  overhead)  are 

needed  to  do  the  work,  whether 
all  required  resources  will  be 
available  when  needed,  and 
whether  any  funding  or  time 
constraints  exist. 


4.  Establishing  The  schedule  should  Substantially 

the  duration  of  realistically  reflect  how  long 
all  activities  each  activity  will  take  to 

execute.  In  determining  the 
duration  of  each  activity,  the 
same  rationale,  historical  data, 
and  assumptions  used  for  cost 
estimating  should  be  used. 

Durations  should  be  as  short 
as  possible  and  have  specific 
start  and  end  dates.  The 
schedule  should  be  continually 
monitored  to  determine  when 
forecasted  completion  dates 
differ  from  planned  dates;  this 
information  can  be  used  to 
determine  whether  schedule 
variances  will  affect 
subsequent  work. 


GAO  analysis 

The  ECSS  PMO  stated  that  the  government  is  aware  that  the 
contractor  assigns  resources  to  activities,  but  the  government 
has  no  detailed  insight  into  the  resources  because  of  the  current 
FFP  contractual  arrangement.  However,  the  program  office  was 
not  able  to  provide  evidence  that  would  confirm  that  the 
schedule  is  resource  loaded.  Resource  information  would  assist 
the  program  office  in  forecasting  the  likelihood  of  activities  being 
completed  based  on  their  projected  end  dates.  If  the  current 
schedule  does  not  allow  for  insight  into  current  or  projected  over¬ 
allocation  of  resources,  then  the  risk  of  the  program  slipping  is 
significantly  increased. 

Eighty-eight  percent  of  remaining  activities  meet  best  practices 
for  durations,  being  less  than  44  days  (or  two  working  months). 
Seventy  activities  (4  percent)  have  longer  than  100-day 
durations;  the  PMO  has  identified  the  majority  of  these  as  level- 
of-effort  support  activities.  Twenty-five  of  these  level-of-effort 
activities  span  the  start  and  end  dates  of  the  project  and  appear 
in  the  schedule  as  critical  activities.  Level-of-effort  activities, 
such  as  systems  engineering  and  program  management,  cannot 
define  the  critical  path  because  they  are  nondiscrete  support 
activities  that  do  not  produce  a  definite  end  product. 
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Best  practice  Explanation 


Criterion  met 


5.  Integrating 
schedule 
activities 
horizontally  and 
vertically 


The  schedule  should  be 
horizontally  integrated, 
meaning  that  it  should  link 
products  and  outcomes 
associated  with  other 
sequenced  activities.  These 
links  are  commonly  referred  to 
as  “handoffs”  and  serve  to 
verify  that  activities  are 
arranged  in  the  right  order  to 
achieve  aggregated  products 
or  outcomes.  The  schedule 
should  also  be  vertically 
integrated,  meaning  that  the 
dates  for  starting  and 
completing  activities  in  the 
integrated  master  schedule 
should  be  aligned  with  the 
dates  for  supporting  tasks  and 
subtasks.  Such  mapping  or 
alignment  among  levels 
enables  different  groups  to 
work  to  the  same  master 
schedule. 


Partially 


6.  Establishing  Scheduling  software  should  be  Partially 
the  critical  path  used  to  identify  the  critical  path, 
for  all  activities  which  represents  the  chain  of 
dependent  activities  with  the 
longest  total  duration. 

Establishing  a  project’s  critical 
path  is  necessary  to  examine 
the  effects  of  any  activity 
slipping  along  this  path. 

Potential  problems  along  or 
near  the  critical  path  should 
also  be  identified  and  reflected 
in  scheduling  the  duration  of 
high-risk  activities. 


GAO  analysis 

We  found  that  vertical  integration — that  is,  the  ability  to 
consistently  trace  work  breakdown  structure  elements  between 
detailed,  intermediate,  and  master  schedules — is  demonstrated 
because  the  overall  ECSS  schedule  is  made  up  of  individual 
project  schedules  like  the  Solutions  Development  schedule. 
However,  issues  with  reliance  on  hard  date  constraints,  the 
overuse  of  lags,  critical  level-of-effort  tasks,  and  instances  of 
convoluted  logic  such  as  start-to-finish  links  keep  this  detailed 
schedule  from  fully  complying  with  the  requirement  of  horizontal 
integration — that  is,  the  overall  ability  of  the  schedule  to  depict 
relationships  between  different  program  elements  and  product 
handoffs.  Horizontal  integration  demonstrates  that  the  overall 
schedule  is  rational,  planned  in  a  logical  sequence,  accounts  for 
interdependencies  between  work  and  planning  packages,  and 
provides  a  way  to  evaluate  current  status. 


Our  analysis  could  not  determine  a  valid  critical  path — the 
longest  duration  path  through  the  sequenced  list  of  activities — 
because  level-of  -effort  activities  define  the  start  and  finish  dates 
of  the  detail  planning  portion  of  the  project.  Level  of  effort 
activities  should  not  drive  the  critical  path  because  they  only 
serve  to  support  detail  work  activities.  The  PMO  acknowledged 
that  a  critical  path  would  be  difficult  to  calculate  within  the 
schedule  because  the  project  schedules  are  team-oriented 
rather  than  product-oriented,  which  causes  complex  linking 
relationships.  While  a  true  critical  path  does  not  exist  throughout 
all  46  project  schedules,  program  management  reviews  a  high- 
level,  manually  constructed  “Critical  Events”  schedule  that  tracks 
the  status  of  major  program  milestones.  These  major  program 
milestones  are  linked  to  lower-level  schedules,  and  their  status 
is  updated  daily  and  reviewed  each  week  by  program 
management.  However,  it  is  important  that  the  lower-level 
schedules  include  complete  logic  that  addresses  the 
relationships  between  predecessor  and  successor  activities, 
because  any  activity  can  become  critical  under  some 
circumstances.  Without  clear  insight  into  a  critical  path  at  the 
project  level,  management  will  not  be  able  to  monitor  critical  or 
near-critical  detail  activities  that  may  have  a  detrimental  impact 
on  downstream  activities  if  delayed. 
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Best  practice 

Explanation 

Criterion  met 

GAO  analysis 

7.  Identifying 
reasonable  float 

The  schedule  should  identify 
the  float — the  amount  of  time 
by  which  a  predecessor  activity 
can  slip  before  the  delay 
affects  successor  activities — so 
that  a  schedule’s  flexibility  can 
be  determined.  As  a  general 
rule,  activities  along  the  critical 
path  have  the  least  float.  Total 
float  is  the  total  amount  of  time 
by  which  an  activity  can  be 
delayed  without  delaying  the 
project’s  completion,  if 
everything  else  goes  according 
to  plan. 

Partially 

Most  remaining  tasks  appear  to  have  reasonable  total  float,  but 
there  are  587  activities  (31  percent)  with  over  50  days  (2  working 
months)  of  total  float.  In  other  words,  according  to  the  schedule, 
587  remaining  activities  (20  percent)  could  be  delayed  by  2 
working  months  and  not  delay  the  final  activity  in  the  Solutions 
Development  schedule.  Activities  with  large  float  values  may 
indicate  a  lack  of  completeness  in  the  schedule  logic.  The  PMO 
stated  that  total  float  is  monitored  by  management  in  higher-level 
milestone  schedules,  not  lower-level  project  schedules.  Incorrect 
float  estimates  will  result  in  an  invalid  critical  path,  and  will  result 
in  an  inability  to  allocate  resources  from  non-critical  activities  to 
activities  that  cannot  slip  without  affecting  the  project  finish  date. 

8.  Conducting  a 
schedule  risk 
analysis 

A  schedule  risk  analysis  should 
be  performed  using  statistical 
techniques  to  predict  the  level 
of  confidence  in  meeting  a 
project's  completion  date.  This 
analysis  focuses  not  only  on 
critical  path  activities  but  also 
on  activities  near  the  critical 
path,  since  they  can  affect  the 
project's  status. 

Not  met 

PMO  officials  stated  that  while  the  program  reviews  the  schedule 
on  a  weekly  basis  and  assesses  risks  to  the  program,  it  has  not 
performed  a  schedule  risk  analysis.  Best  practices  suggest  that 
a  schedule  risk  analysis  can  be  used  to  determine  a  level  of 
confidence  in  meeting  the  completion  date  or  whether  proper 
reserves  have  been  incorporated  into  the  schedule.  Such  an 
analysis  will  calculate  schedule  reserve,  which  can  be  set  aside 
for  those  activities  identified  as  high  risk.  Without  this  reserve, 
the  program  faces  the  risk  of  delays  to  the  scheduled  completion 
date  if  any  delays  were  to  occur  in  critical  path  activities. 

9.  Updating  the 
schedule  using 
logic  and 
durations  to 
determine  the 
dates 

The  schedule  should  be 
continuously  updated  using 
logic  and  durations  to 
determine  realistic  start  and 
completion  dates  for  program 
activities.  The  schedule  should 
be  analyzed  continuously  for 
variances  to  determine  when 
forecasted  completion  dates 
differ  from  planned  dates.  This 
analysis  is  especially  important 
for  those  variations  that  impact 
activities  identified  as  being  in 
a  project’s  critical  path  and  can 
impact  a  scheduled  completion 
date. 

Partially 

The  status  date  for  the  version  of  the  schedule  we  analyzed  is 
January  1 , 201 0,  a  federal  holiday.  A  status  date  denotes  the 
date  of  the  latest  update  to  the  schedule  and  thus  defines  the 
point  in  time  at  which  completed  work  and  remaining  work  are 
calculated.  The  PMO  could  not  confirm  that  this  date  was 
correct.  Assuming  the  status  date  is  correct,  we  found  several 
date  anomalies  within  the  schedule,  suggesting  that 
management  may  need  to  review  how  and  when  the  schedule  is 
updated.  We  found  29  activities  (2  percent)  that  should  have 
started  but  have  no  actual  start  date;  24  activities  (1  percent) 
that  should  have  finished  but  have  no  actual  finish  date;  and  9 
milestone  activities  with  actual  finish  dates  in  the  future.  In 
addition,  we  found  24  instances  (1  percent)  of  out-of-sequence 
logic — that  is,  actual  progress  being  recorded  on  activities  that, 
according  to  schedule  logic,  should  not  have  begun  yet.  This  is  a 
common  occurrence  in  scheduling,  as  actual  events  often 

override  planned  logic.  However,  some  of  these  successor 
activities  are  planned  to  begin  2  to  3  months  in  the  future, 
suggesting  that  the  schedule  logic  should  be  updated  to  reflect 
changes. 


Source:  GAO  analysis  based  on  data  provided  by  the  ECSS  PMO. 

Note:  The  ECSS  program  schedule  consists  of  a  master  schedule  with  46  embedded  project 
schedules  representing  individual  product  teams,  or  workstreams.  The  46  schedules  include  2  high- 
level  schedules,  one  dedicated  to  key  date  milestones  and  another  to  critical  events.  Two  project 
schedules  were  chosen  based  on  their  importance  to  the  program  and  the  high  amount  of  activity 
currently  associated  with  the  product  team. 
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Table  13:  Analysis  of  the  Air  Force’s  ECSS  Reports,  Interfaces,  Conversions,  and  Extensions  (RICE)  Program  Schedule 


Best  practice  Explanation 


Criterion  met  GAO  analysis 


1 .  Capturing  all  The  schedule  should  reflect  all  Substantially 
activities  activities  as  defined  in  the 

project’s  work  breakdown 
structure,  which  defines  in 
detail  the  work  necessary  to 
accomplish  a  project’s 
objectives,  including  activities 
to  be  performed  by  both  the 
owner  and  contractors. 


2.  Sequencing  The  schedule  should  be  Partially 

all  activities  planned  so  that  critical  project 
dates  can  be  met.  To  meet  this 
objective,  activities  need  to  be 
logically  sequenced — that  is, 
listed  in  the  order  in  which  they 
are  to  be  carried  out.  In 
particular,  activities  that  must 
be  completed  before  other 
activities  can  begin 
(predecessor  activities),  as  well 
as  activities  that  cannot  begin 
until  other  activities  are 
completed  (successor 
activities),  should  be  identified. 

This  helps  ensure  that 
interdependencies  among 
activities  that  collectively  lead 
to  the  accomplishment  of 
events  or  milestones  can  be 
established  and  used  as  a 
basis  for  guiding  work  and 
measuring  progress. 


While  the  PMO  does  have  detailed  schedules  of  government 
effort — a  commendable  best  practice — these  are  not  fully 
integrated  into  an  integrated  master  schedule  (IMS)  with  the 
contractor  schedules.  Our  analysis  found  that  the  ECSS  RICE 
schedule  contains  “touch  points,”  or  links  between  government 
and  contractor  activities,  representing  dependencies  between 
contractor  and  government  activities.  However,  the  government 
activities  are  not  completely  linked  to  government  schedules 
maintained  and  updated  by  the  government  PMO. 

Our  analysis  found  that  activities  in  the  RICE  workstream 
schedule  are  mapped  to  contractor  work  breakdown  structure 
elements  and  can  be  traced  to  completion  criteria  and 
descriptions  of  associated  work  products. 

Our  analysis  shows  that  472  of  the  4,433  remaining  activities,  or 
1 1  percent,  have  missing  logic.  Missing  predecessors  or 
successors  are  usually  a  signal  of  broken  logic  and  reduce  the 
credibility  of  the  calculated  dates.  If  an  activity  that  has  no  logical 
successor  slips,  the  schedule  will  not  reflect  the  effect  on  the 
critical  path,  float,  or  scheduled  start  dates  of  future  activities.  In 
addition,  we  found  820  remaining  activities,  or  19  percent,  have 
“dangling”  logic.  Of  these  820  activities  with  dangling  logic,  241 
activities  are  missing  logic  that  would  determine  their  start  dates. 
Because  their  start  dates  are  not  determined  by  logic,  these 
activities  would  have  to  start  earlier  in  order  to  finish  on  time  if 
they  ran  longer  than  their  planned  durations.  The  other  579 
activities  with  dangling  logic  are  missing  successors  off  their 
finish  date.  In  other  words,  these  activities  could  continue 
indefinitely  and  not  affect  the  start  or  finish  dates  of  future 
activities. 

We  found  277  Start  No  Earlier  Than  constraints  (6  percent) 
within  the  schedule.  These  are  considered  “soft”  date  constraints 
in  that  they  allow  the  activity  to  slip  into  the  future  based  on  what 
happens  to  their  predecessor  activities.  Activities  may  be  soft 
constrained,  for  example,  to  represent  receipt  of  delivery  of 
equipment.  However,  in  general  constraining  an  activity's  start 
date  prevents  managers  from  accomplishing  work  as  soon  as 
possible  and  consumes  flexibility  in  the  project. 

Of  the  remaining  activities,  91  activities  (2  percent)  are  linked  to 
their  successor  activities  with  lags,  including  a  lag  greater  than 
1 00  days.  Lags  represent  the  passing  of  time  between  activities 
but  are  often  misused  to  put  activities  on  a  specific  date  or  to 
insert  a  buffer  for  risk.  Lags  should  be  justified  because  they 
cannot  have  risk  or  uncertainty.  Without  logically  sequencing 
activities,  the  schedule  cannot  be  used  as  a  reliable  basis  for 
guiding  work  and  measuring  progress. 
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Best  practice  Explanation  Criterion  met 

3.  Assigning  The  schedule  should  reflect  Minimally 

resources  to  all  what  resources  (e.g.,  labor, 
activities  materials,  and  overhead)  are 

needed  to  do  the  work,  whether 
all  required  resources  will  be 
available  when  needed,  and 
whether  any  funding  or  time 
constraints  exist. 


4.  Establishing  The  schedule  should  Substantially 

the  duration  of  realistically  reflect  how  long 
all  activities  each  activity  will  take  to 

execute.  In  determining  the 
duration  of  each  activity,  the 
same  rationale,  historical  data, 
and  assumptions  used  for  cost 
estimating  should  be  used. 

Durations  should  be  as  short 
as  possible  and  have  specific 
start  and  end  dates.  The 
schedule  should  be  continually 
monitored  to  determine  when 
forecasted  completion  dates 
differ  from  planned  dates;  this 
information  can  be  used  to 
determine  whether  schedule 
variances  will  affect 
subsequent  work. 


5.  Integrating 
schedule 
activities 
horizontally  and 
vertically 


The  schedule  should  be 
horizontally  integrated, 
meaning  that  it  should  link 
products  and  outcomes 
associated  with  other 
sequenced  activities.  These 
links  are  commonly  referred  to 
as  “handoffs”  and  serve  to 
verify  that  activities  are 
arranged  in  the  right  order  to 
achieve  aggregated  products 
or  outcomes.  The  schedule 
should  also  be  vertically 
integrated,  meaning  that  the 
dates  for  starting  and 
completing  activities  in  the 
integrated  master  schedule 
should  be  aligned  with  the 
dates  for  supporting  tasks  and 
subtasks.  Such  mapping  or 
alignment  among  levels 
enables  different  groups  to 
work  to  the  same  master 


Partially 


GAO  analysis 

The  ECSS  PMO  stated  that  the  government  is  aware  that  the 
contractor  assigns  resources  to  activities,  but  the  government 
has  no  detailed  insight  into  the  resources  because  of  the  current 
FFP  contractual  arrangement.  However,  the  program  office  was 
not  able  to  provide  evidence  that  would  confirm  the  schedule  is 
resource  loaded.  Resource  information  would  assist  the  program 
office  in  forecasting  the  likelihood  of  activities  being  completed 
based  on  their  projected  end  dates.  If  the  current  schedule  does 
not  allow  for  insight  into  current  or  projected  over-allocation  of 
resources,  then  the  risk  of  the  program  slipping  is  significantly 
increased. 

Ninety-seven  percent  of  the  remaining  activities  meet  best 
practices  for  durations,  being  less  than  44  days  (or  two  working 
months).  Sixty  activities  (1  percent)  have  longer  than  100-day 
durations,  which  the  PMO  has  identified  as  level-of-effort  support 
activities.  Forty-two  of  these  level-of-effort  activities  span  the 
start  and  end  dates  of  the  project  and  appear  in  the  schedule  as 
critical  activities.  Level-of-effort  activities,  such  as  systems 
engineering  and  program  management,  cannot  define  the  critical 
path  because  they  are  nondiscrete  support  activities  that  do  not 
produce  a  definite  end  product. 


We  found  that  vertical  integration — that  is,  the  ability  to 
consistently  trace  work  breakdown  structure  elements  between 
detailed,  intermediate,  and  master  schedules — is  demonstrated 
because  the  overall  ECSS  schedule  is  made  up  of  individual 
project  schedules  like  the  RICE  schedule.  However,  issues  with 
missing  dependencies,  activities  with  dangling  logic,  overuse  of 
lags,  and  critical  level-of-effort  activities  keep  this  detailed 
schedule  from  being  fully  compliant  with  the  requirement  of 
horizontal  integration — that  is,  the  overall  ability  of  the  schedule 
to  depict  relationships  between  different  program  elements  and 
product  handoffs.  Horizontal  integration  demonstrates  that  the 
overall  schedule  is  rational,  planned  in  a  logical  sequence, 
accounts  for  interdependencies  between  work  and  planning 
packages,  and  provides  a  way  to  evaluate  current  status. 
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Best  practice  Explanation  Criterion  met 

schedule. 

6.  Establishing  Scheduling  software  should  be  Partially 
the  critical  path  used  to  identify  the  critical  path, 
for  all  activities  which  represents  the  chain  of 
dependent  activities  with  the 
longest  total  duration. 

Establishing  a  project’s  critical 
path  is  necessary  to  examine 
the  effects  of  any  activity 
slipping  along  this  path. 

Potential  problems  along  or 
near  the  critical  path  should 
also  be  identified  and  reflected 
in  scheduling  the  duration  of 
high-risk  activities. 


7.  Identifying  The  schedule  should  identify  Partially 
reasonable  float  the  float — the  amount  of  time 

by  which  a  predecessor  activity 
can  slip  before  the  delay 
affects  successor  activities — so 
that  a  schedule's  flexibility  can 
be  determined.  As  a  general 
rule,  activities  along  the  critical 
path  have  the  least  float.  Total 
float  is  the  total  amount  of  time 
by  which  an  activity  can  be 
delayed  without  delaying  the 
project's  completion,  if 
everything  else  goes  according 
to  plan. 

8.  Conducting  a  A  schedule  risk  analysis  should  Not  met 

schedule  risk  be  performed  using  statistical 
analysis  techniques  to  predict  the  level 

of  confidence  in  meeting  a 
project's  completion  date.  This 
analysis  focuses  not  only  on 
critical  path  activities  but  also 
on  activities  near  the  critical 
path,  since  they  can  affect  the 
project's  status. 


GAO  analysis 


Our  analysis  could  not  determine  a  valid  critical  path — the 
longest  duration  path  through  the  sequenced  list  of  activities — 
because  nearly  30  percent  of  remaining  activities  have  missing 
or  incomplete  logic,  and  because  level-of-effort  tasks  (209  days 
long)  define  the  start  and  finish  dates  of  the  project.  Level-of- 
effort  activities  should  not  drive  the  critical  path  because  they 
only  serve  to  support  detail  work  activities.  The  government 
PMO  acknowledged  that  a  critical  path  would  be  difficult  to 
calculate  within  the  schedule  because  the  project  schedules  are 
team-oriented  rather  than  product-oriented,  which  causes 
complex  linking  relationships.  While  a  true  critical  path  does  not 
exist  throughout  all  46  project  schedules,  program  management 
reviews  a  high-level,  manually  constructed  Critical  Events 
schedule  that  tracks  the  status  of  major  program  milestones. 
These  major  program  milestones  are  linked  to  lower-level 
schedules,  and  their  status  is  updated  daily  and  reviewed  each 
week  by  program  management.  However,  it  is  important  that  the 
lower  level  schedules  include  complete  logic  that  addresses  the 
relationships  between  predecessor  and  successor  activities, 
because  any  activity  can  become  critical  under  some 
circumstances.  Without  clear  insight  into  a  critical  path  at  the 
project  level,  management  will  not  be  able  to  monitor  critical  or 
near-critical  detail  activities  that  may  have  a  detrimental  impact 
on  downstream  activities  if  delayed. 

We  found  that  the  schedule  did  not  have  a  reasonable  amount 
of  float  because  78  percent  of  remaining  activities  have  zero 
days  of  total  float.  In  other  words,  according  to  the  schedule, 
3,448  remaining  activities  cannot  slip  one  day  without  delaying 
the  finish  date  of  the  project  by  one  day.  The  program  lead 
scheduler  stated  that  total  float  is  monitored  by  management  at 
the  higher  critical  events  schedules,  not  lower-level  project 
schedules.  However,  incorrect  float  estimates  in  lower-level 
schedules  will  result  in  an  invalid  critical  path,  and  will  result  in 
an  inability  to  allocate  resources  from  non-critical  activities  to 
activities  that  cannot  slip  without  affecting  the  project  finish  date. 


PMO  officials  stated  that  while  the  program  reviews  the  schedule 
on  a  weekly  basis  and  assesses  risks  to  the  program,  it  has  not 
performed  a  schedule  risk  analysis.  Best  practices  suggest  that 
a  schedule  risk  analysis  can  be  used  to  determine  a  level  of 
confidence  in  meeting  the  completion  date  or  to  determine 
whether  proper  reserves  have  been  incorporated  into  the 
schedule.  Such  an  analysis  will  calculate  schedule  reserve, 
which  can  be  set  aside  for  those  activities  identified  as  high  risk. 
Without  this  reserve,  the  program  faces  the  risk  of  delays  to  the 
scheduled  completion  date  if  any  delays  were  to  occur  on  critical 
path  activities. 
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Best  practice  Explanation 


Criterion  met  GAO  analysis 


9.  Updating  the 
schedule  using 
logic  and 
durations  to 
determine  the 
dates 


The  schedule  should  be  Partially 

continuously  updated  using 
logic  and  durations  to 
determine  realistic  start  and 
completion  dates  for  program 
activities.  The  schedule  should 
be  analyzed  continuously  for 
variances  to  determine  when 
forecasted  completion  dates 
differ  from  planned  dates.  This 
analysis  is  especially  important 
for  those  variations  that  impact 
activities  identified  as  being  in 
a  project’s  critical  path  and  can 
impact  a  scheduled  completion 
date. 


The  status  date  for  the  version  of  the  schedule  we  analyzed  is 
January  1 , 201 0,  a  federal  holiday.  A  status  date  denotes  the 
date  of  the  latest  update  to  the  schedule  and  thus  defines  the 
point  in  time  at  which  completed  work  and  remaining  work  are 
calculated.  The  PMO  could  not  confirm  that  this  date  was 
correct.  Assuming  the  status  date  is  correct,  we  found  several 
date  anomalies  within  the  schedule,  suggesting  that 
management  may  need  to  review  how  and  when  the  schedule  is 
updated.  For  example,  we  found  14  activities  (less  than  1 
percent)  that  should  have  started  but  have  no  actual  start  date; 
17  activities  (less  than  1  percent)  that  should  have  finished  but 
have  no  actual  finish  date;  and  155  activities  (3  percent)  that 
occurred  in  the  past  according  to  the  schedule  but  are  missing 
both  actual  start  dates  and  actual  finish  dates. 

In  addition,  we  found  22  (less  than  1  percent)  instances  of  out- 
of-sequence  logic — that  is,  actual  progress  being  recorded  on 
activities  that,  according  to  schedule  logic,  should  not  have 
begun  yet.  This  is  a  common  occurrence  in  scheduling,  as  actual 
events  often  override  planned  logic.  However,  schedule  logic 
should  be  updated  to  reflect  changes  as  much  as  possible. 


Source:  GAO  analysis  based  on  data  provided  by  the  ECSS  PMO. 


Page  84 


GAO-11-53  DOD  Business  Systems 


Appendix  IV:  Assessments  of  Four  DOD  ERP 
Programs’  Integrated  Master  Schedules 


Table  14:  Analysis  of  the  Army’s  GFEBS  Program  Schedule 


Best  practice  Explanation 


Initial  result  Final  result  GAO  analysis 


1 .  Capturing  all 
activities 


2.  Sequencing 
all  activities 


The  schedule  should  reflect  Substantially  Substantially 

all  activities  as  defined  in  the 

project’s  work  breakdown 

structure,  which  defines  in 

detail  the  work  necessary  to 

accomplish  a  project’s 

objectives,  including  activities 

to  be  performed  by  both  the 

owner  and  contractors. 


The  schedule  should  be  Minimally  Partially 

planned  so  that  critical  project 

dates  can  be  met.  To  meet 

this  objective,  activities  need 

to  be  logically  sequenced — 

that  is,  listed  in  the  order  in 

which  they  are  to  be  carried 

out.  In  particular,  activities 

that  must  be  completed 

before  other  activities  can 

begin  (predecessor  activities), 

as  well  as  activities  that 

cannot  begin  until  other 

activities  are  completed 

(successor  activities),  should 

be  identified.  This  helps 

ensure  that 

interdependencies  among 
activities  that  collectively  lead 
to  the  accomplishment  of 
events  or  milestones  can  be 
established  and  used  as  a 
basis  for  guiding  work  and 


Initial  Analysis:  Our  analysis  found  that  while  the  Wave 
4  deployment  schedule  captures  both  contractor  and 
government  activities,  the  program  schedule  is  not 
fully  integrated  because  individual  deployment 
schedules  for  software  releases  are  not  related  to 
activities  within  other  program  schedules.  PMO 
officials  stated  that  the  while  release  and  maintenance 
activities  are  integrated  together  in  one  schedule,  and 
each  deployment  wave  has  its  own  schedule,  the 
schedules  are  not  linked  to  each  other  because  the 
activities  within  each  schedule  are  not  related. 
However,  a  fully  integrated  master  schedule  would  link 
government  and  contractor  development,  deployment, 
and  subsequent  maintenance  activities. 

Activities  in  the  program  schedule  are  mapped  to  the 
program’s  integrated  master  plan,  and  deliverables  in 
the  Wave  4  schedule  are  mapped  to  the  program's 
Quality  Assurance  Surveillance  Plan  through  unique 
identification  numbers.  A  large  portion  of  the  Wave  4 
deployment  schedule  is  made  up  of  receiver 
milestones;  that  is,  products  the  program  needs  to 
receive  from  external  field  sites  before  certain 
activities  can  be  conducted.  In  addition  to  including 
government  and  contractor  activities,  the  schedule 
also  include  tasks  representing  work  being  performed 
by  external  organizations. 

Updated  analysis:  No  change  to  initial  assessment. 

Initial  analysis:  Our  analysis  found  18  activities  of 
2,150  remaining  (less  than  1  percent)  within  the 
schedule  that  have  no  successor  links,  and  three 
activities  (less  than  1  percent)  that  have  neither 
successor  nor  predecessor  links.  Activities  without 
successor  links  do  not  affect  any  other  future  activity. 
That  is,  they  can  continue  until  the  end  of  the  project 
without  affecting  the  finish  date  of  the  project. 

The  schedule  includes  24  (1  percent)  Must  Start  On 
(MSO)  constraints.  An  MSO  constraint  is  considered  a 
“hard”  date  constraint  because  it  prevents  the  activity 
from  starting  earlier  or  later  than  its  planned  date.  This 
renders  the  schedule  rigid  and  prevents  the  schedule 
from  being  dynamic.  An  MSO  constraint  is  artificial 
and  makes  the  scheduled  activity  appear  to  be  on 
track  to  finish  on  time  when  it  may  not  be.  PMO 
schedulers  told  us  that  of  the  24  MSO-constrained 
tasks,  15  (less  than  1  percent  of  all  remaining)  are 
associated  with  executive  briefings  that  are  now  out  of 
scope  and  should  be  removed  from  the  schedule.  Of 
the  remaining  9  MSO-constrained  tasks,  8  are  used  to 
force  successor  activities  to  start  on  exactly  the  first 
days  of  calendar  months.  While  these  constraints  may 
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measuring  progress.  make  scheduling  activities  simpler,  they  have  an 

adverse  effect  on  the  project’s  critical  path.  An  activity 
with  an  MSO  constraint  automatically  becomes  critical 
within  scheduling  software  regardless  of  whether  it 
actually  should  be  critical.  A  final  MSO  constraint  is 
attached  to  the  “Go  Live”  milestone,  which  prevents 
the  project  finish  milestone  from  shifting  because  of 
completed  or  remaining  effort  on  predecessor 
activities.  PMO  officials  acknowledged  that  the  MSO 
constraint  should  not  have  been  applied  to  the  finish 
milestone  and  stated  that  it  would  be  removed  in  the 
next  update  of  the  schedule. 

Our  analysis  also  found  that  50  summary  tasks 
(12  percent  of  remaining  summary  tasks)  have 
predecessor  links.  PMO  schedulers  told  us  that  these 
summary  links  are  used  in  lieu  of  linking  predecessors 
to  the  numerous  lower-level  tasks.  Because  many  of 
the  lower-level  tasks  begin  on  the  same  date,  this 
makes  updating  the  schedule  simpler:  an  updated 
start  date  for  the  summary  task  will  force  that  same 
date  on  all  the  unlinked  lower-level  tasks.  While  this 
indeed  makes  updating  easier,  this  technique  is  not 
considered  a  best  practice.  First,  summary  tasks  do 
not  represent  work  and  are  simply  used  as  grouping 
elements.  As  such,  they  should  take  their  start  and 
finish  dates  from  lower-level  activities;  they  should  not 
dictate  the  start  or  finish  of  lower-level  activities. 
Secondly,  linking  summary  tasks  obfuscates  the  logic 
of  the  schedule.  That  is,  tracing  logic  through 
summary  links  does  not  impart  to  management  the 
sequence  in  which  lower-level  activities  should  be 
carried  out. 

Our  analysis  found  that  358  activities  (17  percent)  are 
scheduled  to  occur  on  a  Sunday.  This  is  a 
consequence  of  a  summary  task  linked  to  a 
constrained  milestone — constrained  to  start  on  the  first 
day  of  a  calendar  month,  which  happened  to  be  a 
Sunday  and  in  turn  causes  a  multitude  of  lower-level 
activities  to  also  begin  on  a  Sunday.  PMO  schedulers 
acknowledged  that  this  was  an  error  and  the  activities 
would  be  shifted  to  begin  on  a  work  day. 

There  are  67  remaining  activities  (3  percent)  that  are 
linked  to  their  successor  activities  with  lags  and  38  (2 
percent)  are  linked  with  negative  lags  (or  “leads”). 

Lags  represent  the  passing  of  time  between  activities 
but  are  often  misused  to  put  activities  on  a  specific 
date  or  to  insert  a  buffer  for  risk.  Lags  should  be 
justified  because  they  cannot  have  risk  or  uncertainty. 
Without  logically  sequenced  activities,  the  schedule 
cannot  be  used  as  a  reliable  basis  for  guiding  work 
and  measuring  progress. 

Updated  analysis:  There  are  still  18  tasks  within  the 

_ schedule  without  successors  (or  less  than  1  percent  of 
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remaining  activities  in  the  updated  schedule).  While 
these  activities  have  finish  dates  in  December  2009 
and  February  2010,  they  do  not  have  actual  finish 
dates  and  we  therefore  cannot  determine  if  these 
activities  are  completed  or  have  the  potential  to  cause 
future  activities  to  slip.  The  updated  schedule  now 
contains  7  MSO  constraints  (less  than  1  percent):  the 
15  constraints  associated  with  executive  meetings 
have  been  removed;  the  MSO  constraint  on  the  “Go 
Live”  milestone  has  been  removed;  2  MSO  constraints 
marking  the  beginning  of  months  have  occurred;  and  1 
new  constraint  has  been  added  to  mark  the  beginning 
of  the  month  following  deployment.  The  358  activities 
unintentionally  scheduled  to  begin  on  a  Sunday  have 
been  altered  by  a  1-day  lag  to  begin  on  a  proper 
workday.  However,  lags  should  not  be  used  in  lieu  of 
logic  to  force  activities  to  start  on  a  specified  date. 
Additionally,  the  updated  schedule  corrects  minor 
missing  predecessor  logic  issues. 


3.  Assigning 
resources  to  all 
activities 


The  schedule  should  reflect 
what  resources  (e.g.,  labor, 
materials,  and  overhead)  are 
needed  to  do  the  work, 
whether  all  required 
resources  will  be  available 
when  needed,  and  whether 
any  funding  or  time 
constraints  exist. 


Not  met  Not  met  Initial  analysis:  GFEBS  officials  stated  that  because  of 

the  current  FFP  contractual  arrangement,  the 
government  does  not  have  insight  into  the  contractor's 
efforts  to  assign  resources  to  activities.  They  stated 
that  while  they  are  aware  that  activities  in  previous 
schedule  releases  were  assigned  resources  by  the 
contractor,  the  current  schedule  is  not  resource 
loaded.  Resource  information  would  assist  the 
program  office  in  forecasting  the  likelihood  of  activities 
being  completed  based  on  their  projected  end  dates.  If 
the  current  schedule  does  not  allow  for  insight  into 
current  or  projected  over-allocation  of  resources,  then 
the  risk  of  the  program  slipping  is  significantly 
increased. 

Updated  analysis:  No  change  to  initial  assessment. 


4.  Establishing  The  schedule  should  Fully  met  Fully  met  Initial  analysis:  Seventy-two  percent  of  remaining 


the  duration  of  realistically  reflect  how  long 
all  activities  each  activity  will  take  to 

execute.  In  determining  the 
duration  of  each  activity,  the 
same  rationale,  historical 
data,  and  assumptions  used 
for  cost  estimating  should  be 
used.  Durations  should  be  as 
short  as  possible  and  have 
specific  start  and  end  dates. 
The  schedule  should  be 
continually  monitored  to 
determine  when  forecasted 
completion  dates  differ  from 
planned  dates;  this 
information  can  be  used  to 
determine  whether  schedule 
variances  will  affect 


activities  meet  best  practices  for  duration,  being  less 
than  44  days  (or  2  working  months).  Activities  with 
excessive  durations  (more  than  100  days)  represent 
effort  being  performed  by  organizations  outside  of  the 
program  office.  Representing  effort  in  the  schedule 
that  is  performed  by  outside  organizations  is 
considered  a  best  practice  because  it  keeps 
management  informed  of  ongoing  work  that  might 
easily  be  forgotten  until  the  deliverable  is  due,  and  the 
impact  on  future  activities  if  the  deliverable  is  behind 
schedule. 

Updated  analysis:  No  change  to  initial  assessment. 
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subsequent  work. 


5.  Integrating 
schedule 
activities 
horizontally  and 
vertically 


The  schedule  should  be 
horizontally  integrated, 
meaning  that  it  should  link 
products  and  outcomes 
associated  with  other 
sequenced  activities.  These 
links  are  commonly  referred 
to  as  “handoffs”  and  serve  to 
verify  that  activities  are 
arranged  in  the  right  order  to 
achieve  aggregated  products 
or  outcomes.  The  schedule 
should  also  be  vertically 
integrated,  meaning  that  the 
dates  for  starting  and 
completing  activities  in  the 
integrated  master  schedule 
should  be  aligned  with  the 
dates  for  supporting  tasks 
and  subtasks.  Such  mapping 
or  alignment  among  levels 
enables  different  groups  to 
work  to  the  same  master 
schedule. 


Minimally 


Minimally 


6.  Establishing  Scheduling  software  should  Minimally  Partially 
the  critical  path  be  used  to  identify  the  critical 
for  all  activities  path,  which  represents  the 
chain  of  dependent  activities 
with  the  longest  total  duration. 

Establishing  a  project’s 
critical  path  is  necessary  to 
examine  the  effects  of  any 
activity  slipping  along  this 
path.  Potential  problems 
along  or  near  the  critical  path 
should  also  be  identified  and 
reflected  in  scheduling  the 
duration  of  high-risk  activities. 


GAO  analysis 


Initial  analysis:  The  GFEBS  program  schedule 
includes  detailed  information  on  release,  deployment, 
and  maintenance  government  and  contractor 
activities.  However,  our  analysis  of  the  schedule 
concludes  that  vertical  integration — that  is,  the  ability 
to  consistently  trace  work  breakdown  structure 
elements  between  detailed,  intermediate,  and  master 
schedules — is  not  fully  demonstrated  because  none  of 
the  activities  within  the  deployment  schedules  are 
related  to  activities  associated  with  release, 
maintenance,  or  other  wave  schedules.  PMO  officials 
stated  that  while  the  release  and  maintenance 
activities  are  integrated  together  in  one  schedule,  and 
each  deployment  wave  has  its  own  schedule,  the 
schedules  are  not  linked  to  each  other  because  the 
activities  within  each  schedule  are  not  related. 
However,  it  is  unlikely  that  deployment  activities  are 
unrelated  to  release  or  maintenance  activities.  Without 
vertically  integrating  the  schedules,  lower-level 
schedules  cannot  be  clearly  traced  to  upper-tiered 
milestones. 

Issues  with  reliance  on  hard  date  constraints,  lags, 
and  instances  of  convoluted  logic  such  as  linked 
summary  tasks,  keep  the  schedule  from  fully 
complying  with  the  requirement  of  horizontal 
integration — that  is,  the  overall  ability  of  the  schedule 
to  clearly  depict  relationships  between  different 
program  elements  and  product  handoffs.  Horizontal 
integration  demonstrates  that  the  overall  schedule  is 
rational,  planned  in  a  logical  sequence,  accounts  for 
interdependencies  between  work  and  planning 
packages,  and  provides  a  way  to  evaluate  current 
status. 

Updated  analysis:  No  change  to  initial  assessment. 

Initial  analysis:  Our  analysis  could  not  determine  a 
valid  critical  path — the  longest  duration  path  through 
the  sequenced  list  of  activities — because  the  “Go  Live” 
finish  milestone  is  constrained  with  an  MSO  constraint. 
An  MSO  constraint  is  considered  a  “hard”  date 
constraint  because  it  prevents  the  activity  from  starting 
earlier  or  later  than  its  planned  date.  This  renders  the 
schedule  rigid  and  prevents  the  schedule  from  being 
dynamic.  An  MSO  constraint  is  artificial  and  makes  the 
scheduled  activity  appear  to  be  on  track  to  finish  on 
time  when  it  may  not  be.  When  the  constraint  is 
removed,  the  “Go  Live”  milestone  slips  two  months 
from  its  constrained  date  of  January  3,  201 1  to 
March  7,  201 1 . 

In  addition,  our  analysis  found  that  without  the  MSO 
constraint,  the  nearest  driving  activity  to  the  “Go  Live” 
milestone  (that  is,  the  activity  determining  the  date  of 
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7.  Identifying  The  schedule  should  identify  Minimally  Minimally 
reasonable  float  the  float — the  amount  of  time 
by  which  a  predecessor 
activity  can  slip  before  the 
delay  affects  successor 
activities — so  that  a 
schedule’s  flexibility  can  be 
determined.  As  a  general 
rule,  activities  along  the 
critical  path  have  the  least 
float.  Total  float  is  the  total 
amount  of  time  by  which  an 
activity  can  be  delayed 
without  delaying  the  project's 
completion,  if  everything  else 
goes  according  to  plan. 


8.  Conducting  a  A  schedule  risk  analysis  Not  met  Not  met 
schedule  risk  should  be  performed  using 

analysis  statistical  techniques  to 

predict  the  level  of  confidence 
in  meeting  a  project’s 
completion  date.  This 

_ analysis  focuses  not  only  on _ 


GAO  analysis 

the  “Go  Live”  activity)  is  in  October  2010.  In  other 
words,  according  to  the  schedule,  no  activity  starting  in 
November  or  December  2010  is  critical  to  determining 
the  “Go  Live”  date.  Without  clear  insight  into  a  critical 
path  at  the  project  level,  management  will  not  be  able 
to  monitor  critical  or  near-critical  detail  activities  that 
may  have  a  detrimental  impact  on  downstream 
activities  if  delayed. 

Updated  analysis:  The  updated  schedule  has  altered 
the  predecessors  on  the  “Go  Live”  finish  milestone  and 
the  MSO  constraint  has  been  removed.  The  “Go  Live” 
date  is  now  scheduled  to  occur  in  February  201 1 . 
However,  the  critical  path,  as  measured  by  the  path 
with  the  lowest  available  float,  shows  only  five 
activities  from  the  “Go  Live”  date  in  February  201 1  to 
the  “Site  Visit  Activities  Complete”  milestone 
completed  in  March  2010.  Because  so  few  activities 
are  on  the  current  critical  path,  no  activities  scheduled 
within  1  or  2  months  of  deployment  are  currently 
driving  the  project  finish  date.  In  addition,  the  earliest 
critical  activity  on  the  path  appears  to  be  a  functional 
survey  scheduled  for  April  1 , 201 0,  that  has  yet  to 
actually  start. 

Initial  analysis:  We  found  that  the  Wave  4  Deployment 
schedule  displays  unrealistic  total  float  values.  For 
example,  1,273  activities  (59  percent)  within  the 
schedule  are  showing  negative  float.  That  is,  these 
activities  are  one  to  242  days  behind  schedule.  Other 
tasks  display  an  unrealistic  amount  of  positive  float:  49 
tasks  (59  percent)  are  showing  100  to  more  than  300 
days  of  total  float.  In  other  words,  according  to  the 
schedule,  49  remaining  activities  could  be  delayed  by 
more  than  4  working  months  and  not  delay  the  final 
activity  in  the  Wave  4  schedule.  As  a  general  rule, 
activities  along  the  critical  path  have  the  least  amount 
of  float.  Activities  with  large  float  values  may  indicate 
some  lack  of  completeness  in  the  schedule  logic. 
Incorrect  float  estimates  will  result  in  an  invalid  critical 
path  and  an  inability  to  allocate  resources  from 
noncritical  activities  to  activities  that  cannot  slip 
without  affecting  the  project  finish  date. 

Updated  analysis:  The  updated  schedule  continues  to 
reflect  unrealistic  float.  For  example,  172  remaining 
activities  (8  percent)  have  from  90  to  252  days  of 
negative  float,  while  25  remaining  activities  (1  percent) 
have  1 04  to  31 0  days  of  float. 

Initial  analysis:  The  PMO  has  not  performed  a 
schedule  risk  analysis.  GFEBS  officials  stated  that 
while  schedule  risks  have  been  discussed  in  team 
meetings,  the  PMO  has  not  performed  a  formal 
schedule  risk  analysis.  However,  officials  stated  that 
they  are  open  to  improving  in  the  area  of  schedule  risk 
analysis.  Best  practices  suggest  that  a  schedule  risk 
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critical  path  activities  but  also 
on  activities  near  the  critical 
path,  since  they  can  affect  the 
project's  status. 


9.  Updating  the 
schedule  using 
logic  and 
durations  to 
determine  the 
dates 


The  schedule  should  be  Minimally 
continuously  updated  using 
logic  and  durations  to 
determine  realistic  start  and 
completion  dates  for  program 
activities.  The  schedule 
should  be  analyzed 
continuously  for  variances  to 
determine  when  forecasted 
completion  dates  differ  from 
planned  dates.  This  analysis 
is  especially  important  for 
those  variations  that  impact 
activities  identified  as  being  in 
a  project’s  critical  path  and 
can  impact  a  scheduled 
completion  date. 


Partially 


GAO  analysis 

analysis  can  be  used  to  determine  a  level  of 
confidence  in  meeting  the  completion  date  or  to 
determine  whether  proper  reserves  have  been 
incorporated  into  the  schedule.  Such  an  analysis  will 
calculate  schedule  reserve,  which  can  be  set  aside  for 
those  activities  identified  as  high-risk.  Without  this 
reserve,  the  program  faces  the  risk  of  delays  to  the 
scheduled  completion  date  if  any  delays  were  to  occur 
on  critical  path  activities. 

Updated  analysis:  No  change  to  initial  assessment. 

Initial  analysis:  The  status  date  for  the  version  of  the 
Wave  4  schedule  we  analyzed  is  May  3,  201 0.  A 
status  date  denotes  the  date  of  the  latest  update  to  the 
schedule  and  thus  defines  the  point  in  time  at  which 
completed  work  and  remaining  work  are  calculated.  As 
of  this  date,  we  found  a  relatively  large  number  of  date 
anomalies  within  the  schedule,  suggesting  that 
management  may  need  to  review  how  and  when  the 
schedule  is  updated.  For  example,  we  found  247 
activities  (1 1  percent)  that  should  have  started  but 
have  no  actual  start  date  and  200  activities  (9  percent) 
that  should  have  finished  but  have  no  actual  finish 
date.  Moreover,  we  found  7  activities  (less  than  1 
percent)  that  have  actual  finish  dates  in  the  future. 
Schedule  logic  should  be  updated  to  reflect  actual 
progress  so  that  management  is  aware  of  the  latest 
plan  and  the  impacts  to  the  project  if  activity  planned 
dates  are  not  met. 

Updated  analysis:  The  updated  schedule  no  longer 
includes  activities  that  have  actual  finish  dates  beyond 
the  status  date.  However,  the  schedule  contains  44 
activities  (2  percent)  that  should  have  started  but  have 
no  actual  start  date;  22  activities  (1  percent)  that 
should  have  finished  but  have  no  actual  finish  date; 
and  109  activities  (5  percent)  that  should  have  started 
and  finished  but  have  neither  an  actual  start  nor  actual 
finish  date. 


Source:  GAO  analysis  based  on  data  provided  by  the  GFEBS  PMO. 

Note:  The  initial  analysis  reflects  our  assessment  of  the  schedule  originally  submitted  by  the  GFEBS 
PMO  for  our  review.  In  response  to  limitations  that  we  identified  and  shared  with  the  GFEBS  PMO, 
the  program  office  enacted  several  formal  changes  to  their  existing  schedule.  The  updated  analysis 
reflects  our  review  of  the  revised  schedule. 
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Table  15:  Analysis  of  the  Army’s  GCSS-Army  Program  Schedule 


Best  practice  Explanation 


Criterion  met  GAO  analysis 


1 .  Capturing  all  The  schedule  should  reflect  all  Partially 
activities  activities  as  defined  in  the 

project’s  work  breakdown 
structure,  which  defines  in 
detail  the  work  necessary  to 
accomplish  a  project’s 
objectives,  including  activities 
to  be  performed  by  both  the 
owner  and  contractors. 


We  found  that  the  GCSS-Army  program  schedule  is  not  fully 
integrated.  While  the  program  schedule  contains  detailed 
contractor  activities,  it  only  contains  some  major  government 
milestones.  Other  government  activities,  such  as  testing  events 
and  future  milestones  beyond  December  2010,  are  displayed  in 
isolated,  high-level  illustrated  documents  rather  than  in  dynamic 
scheduling  documents. 

We  also  found  that  contractor  activities  within  the  program 
schedule  are  assigned  contractor  work  package  numbers  and 
can  be  traced  to  individual  control  account  plans  and  contractor 
work  breakdown  structure  elements.  Activities  are  also  assigned 
integrated  product  teams  and  individual  control  account 
managers. 

However,  without  fully  integrating  government  activities  with 
contractor  activities,  DOD  cannot  guarantee  the  schedule  has 
either  adequately  captured  all  key  activities  necessary  for  the 
program’s  completion  or  that  it  can  reliably  estimate  the  finish 
date  for  the  program. 
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Best  practice  Explanation  Criterion  met 

2.  Sequencing  The  schedule  should  be  Partially 

all  activities  planned  so  that  critical  project 
dates  can  be  met.  To  meet  this 
objective,  activities  need  to  be 
logically  sequenced — that  is, 
listed  in  the  order  in  which  they 
are  to  be  carried  out.  In 
particular,  activities  that  must 
be  completed  before  other 
activities  can  begin 
(predecessor  activities),  as  well 
as  activities  that  cannot  begin 
until  other  activities  are 
completed  (successor 
activities),  should  be  identified. 

This  helps  ensure  that 
interdependencies  among 
activities  that  collectively  lead 
to  the  accomplishment  of 
events  or  milestones  can  be 
established  and  used  as  a 
basis  for  guiding  work  and 
measuring  progress. 


3.  Assigning  The  schedule  should  reflect  Substantially 
resources  to  all  what  resources  (e.g.,  labor, 
activities  materials,  and  overhead)  are 

needed  to  do  the  work,  whether 
all  required  resources  will  be 
available  when  needed,  and 
whether  any  funding  or  time 
constraints  exist. 


GAO  analysis 

We  found  that  only  2  of  2,255  activities  (less  than  1  percent)  are 
missing  dependencies  and  19  activities  (less  than  1  percent) 
have  “dangling”  logic — that  is,  activities  whose  start  or  finish 
dates  are  missing  logic.  These  activities  with  dangling  logic  have 
no  successor  from  their  finish  date,  meaning  they  can  carry  on 
indefinitely  without  affecting  the  start  date  of  any  other  activity. 
While  dependencies  within  the  schedule  are  generally  sound,  60 
percent  of  the  activities  (1 ,360)  have  Start  No  Earlier  Than 
constraints.  Start  No  Earlier  Than  constraints  are  considered 
“soft”  date  constraints  because  they  allow  an  activity  to  slip  into 
the  future  if  their  predecessor  activity  is  delayed,  but  the  activity 
cannot  begin  earlier  than  its  constraint  date.  Program  officials 
stated  that  Start  No  Earlier  Than  constraints  are  used  to 
manually  allocate  resources  and  to  coordinate  data  tests,  which 
rely  on  coordination  with  outside  partners.  Officials  further  stated 
that  individual  control  account  managers  monitor  these 
constraints.  However,  we  found  that  87  percent  of  the 
constraints  were  actively  affecting  the  start  date  of  their 
activities.  That  is,  without  the  constraint,  the  activity  may  be  able 
to  start  sooner.  If  these  activities  cannot  start  earlier,  then  their 
dates  and  dependencies  should  be  updated  to  reflect  reality. 
Constraining  over  half  of  all  activities  to  start  on  or  after  specific 
dates  defeats  the  purpose  of  a  dynamic  scheduling  tool  and 
greatly  reduces  to  ability  of  the  program  to  take  advantage  of 
possible  time  savings. 

We  also  found  143  Finish  No  Earlier  Than  constraints  (6 
percent).  These  are  also  considered  “soft”  date  constraints 
because  they  prevent  activities  from  finishing  earlier  than  their 
constraint  date.  Program  officials  stated  that  these  were 
erroneously  created  in  the  schedule  during  an  internal  file 
conversion  process  and  would  be  removed  in  the  next  version  of 
the  program  schedule.  Without  logically  sequenced  activities,  the 
schedule  cannot  be  used  as  a  reliable  basis  for  guiding  work  and 
measuring  progress. 

While  the  integrated  master  schedule  is  not  resource  loaded, 
scheduled  activities  can  be  traced  to  control  account  plans  which 
have  resources  laid  out  by  month  by  labor  category.  Budgets  are 
assigned  at  the  control  account  level  and  resources  are 
accounted  for  in  monthly  updates  to  the  program’s  earned  value 
management  system. 
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Best  practice  Explanation  Criterion  met 

4.  Establishing  The  schedule  should  Fully  met 

the  duration  of  realistically  reflect  how  long 
all  activities  each  activity  will  take  to 

execute.  In  determining  the 
duration  of  each  activity,  the 
same  rationale,  historical  data, 
and  assumptions  used  for  cost 
estimating  should  be  used. 

Durations  should  be  as  short 
as  possible  and  have  specific 
start  and  end  dates.  The 
schedule  should  be  continually 
monitored  to  determine  when 
forecasted  completion  dates 
differ  from  planned  dates;  this 
information  can  be  used  to 
determine  whether  schedule 
variances  will  affect 
subsequent  work. 


5.  Integrating 
schedule 
activities 
horizontally  and 
vertically 


The  schedule  should  be 
horizontally  integrated, 
meaning  that  it  should  link 
products  and  outcomes 
associated  with  other 
sequenced  activities.  These 
links  are  commonly  referred  to 
as  “handoffs”  and  serve  to 
verify  that  activities  are 
arranged  in  the  right  order  to 
achieve  aggregated  products 
or  outcomes.  The  schedule 
should  also  be  vertically 
integrated,  meaning  that  the 
dates  for  starting  and 
completing  activities  in  the 
integrated  master  schedule 
should  be  aligned  with  the 
dates  for  supporting  tasks  and 
subtasks.  Such  mapping  or 
alignment  among  levels 
enables  different  groups  to 
work  to  the  same  master 
schedule. 


Partially 


GAO  analysis 

Ninety-eight  percent  of  remaining  activities  meet  the  best 
practice  for  activity  duration,  being  less  than  44  days.  Only  two 
remaining  activities  have  durations  that  exceed  the  best  practice, 
extending  beyond  80  days. 


The  schedule  is  vertically  integrated,  with  low-level  tasks  and 
milestones  being  traceable  to  higher-level  summary  tasks.  While 
the  schedule  has  a  relatively  small  number  of  missing 
dependencies  and  activities  with  dangling  logic,  the  use  of  date 
constraints  on  more  than  60  percent  of  remaining  activities, 
prevent  the  schedule  from  being  completely  horizontally 
integrated.  That  is,  the  date  constraints  limit  the  overall  ability  of 
the  schedule  to  depict  dynamic  relationships  between  different 
program  elements  and  product  handoffs.  Horizontal  integration 
demonstrates  that  the  overall  schedule  is  rational,  planned  in  a 
logical  sequence,  accounts  for  interdependencies  between  work 
and  planning  packages,  and  provides  a  way  to  evaluate  current 
status. 
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Best  practice  Explanation  Criterion  met 

6.  Establishing  Scheduling  software  should  be  Partially 
the  critical  path  used  to  identify  the  critical  path, 
for  all  activities  which  represents  the  chain  of 
dependent  activities  with  the 
longest  total  duration. 

Establishing  a  project’s  critical 
path  is  necessary  to  examine 
the  effects  of  any  activity 
slipping  along  this  path. 

Potential  problems  along  or 
near  the  critical  path  should 
also  be  identified  and  reflected 
in  scheduling  the  duration  of 
high-risk  activities. 


7.  Identifying  The  schedule  should  identify  Substantially 
reasonable  float  the  float — the  amount  of  time 
by  which  a  predecessor  activity 
can  slip  before  the  delay 
affects  successor  activities — so 
that  a  schedule's  flexibility  can 
be  determined.  As  a  general 
rule,  activities  along  the  critical 
path  have  the  least  float.  Total 
float  is  the  total  amount  of  time 
by  which  an  activity  can  be 
delayed  without  delaying  the 
project's  completion,  if 
everything  else  goes  according 
to  plan. 


GAO  analysis 

We  found  that  a  reliable  and  realistic  critical  path  could  not  be 
determined  within  the  program  schedule,  and  program  officials 
agreed  with  our  assessment.  Program  officials  stated  that  the 
schedule  is  constructed  to  increase  the  visibility  of  each  software 
object’s  development,  and  as  a  consequence  of  this  amount  of 
detail,  a  critical  path  cannot  be  shown.  The  schedule  displays 
the  detailed  development  life  cycle  for  hundreds  of  objects  and 
depending  on  the  order  in  which  objects  are  completed,  the 
dependencies  between  objects,  and  the  constant  reallocation  of 
resources,  a  traditional  critical  path  may  be  too  volatile  to  be 
useful.  However,  officials  stated  that  a  higher-level  summary 
type  schedule,  which  would  display  a  valid  critical  path,  would 
not  allow  management  the  proper  insight  into  the  risks 
underlying  the  development  of  each  object.  In  lieu  of  a  traditional 
critical  path,  program  management  monitors  object  development 
weekly  and  program  officials  stated  that  they  are  fully  aware  of 
which  activities  are  behind  or  ahead  of  schedule. 

It  is  commendable  that  the  schedule  includes  the  necessary 
amount  of  complexity  and  detail  to  track  lower-level,  high-risk 
development  activities.  However,  our  analysis  found  that  a 
critical  path  could  not  be  derived  because  of  artificial  date 
constraints  rather  than  complex  object  development  detail. 
Program  officials  stated  that  three  major  milestones  are  being 
tracked:  Critical  Design  Review,  DTOE  1.1,  and  Build/Design 
Phase  Completion.  We  found  that  critical  paths  do  not  exist  for 
any  of  these  milestones  because  of  artificial  date  constraints  on 
activities  unrelated  to  detailed  object  development.  As  a  result, 
we  cannot  determine  a  critical  path  to  any  major  milestone 
based  on  actual  effort  related  to  object  development.  In  this 
respect,  the  schedule  cannot  reliably  forecast  completion  dates 
for  Critical  Design  Review,  DTOE,  Build/Design  Completion,  or, 
as  a  consequence,  Milestone  C. 

The  majority  of  remaining  tasks  in  the  GCSS-Army  contractor 
schedule  appear  to  have  reasonable  total  float  values,  varying 
from  0  to  30  days.  Program  office  officials  stated  that  they 
believe  the  schedule  portrays  accurate  float.  However,  our 
analysis  found  338  (15  percent)  remaining  activities  with  over 
100  days  of  total  float.  In  other  words,  according  to  the  schedule, 
338  remaining  activities  could  be  delayed  by  4  months  and  not 
delay  the  final  project  date.  Activities  with  large  float  values  may 
indicate  a  lack  of  completeness  in  the  schedule  logic. 
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Best  practice  Explanation 


Criterion  met 


8.  Conducting  a 
schedule  risk 
analysis 


A  schedule  risk  analysis  should  Minimally 
be  performed  using  statistical 
techniques  to  predict  the  level 
of  confidence  in  meeting  a 
project's  completion  date.  This 
analysis  focuses  not  only  on 
critical  path  activities  but  also 
on  activities  near  the  critical 
path,  since  they  can  affect  the 
project's  status. 


9.  Updating  the 
schedule  using 
logic  and 
durations  to 
determine  the 
dates 


The  schedule  should  be  Substantially 

continuously  updated  using 

logic  and  durations  to 

determine  realistic  start  and 

completion  dates  for  program 

activities.  The  schedule  should 

be  analyzed  continuously  for 

variances  to  determine  when 

forecasted  completion  dates 

differ  from  planned  dates.  This 

analysis  is  especially  important 

for  those  variations  that  impact 

activities  identified  as  being  in 

a  project’s  critical  path  and  can 

impact  a  scheduled  completion 

date. 


GAO  analysis 

Program  office  officials  stated  that  a  schedule  risk  analysis  is  not 
routinely  performed,  and  that  there  is  currently  no  requirement 
for  the  contractor  to  do  so.  While  no  detailed  risk  analysis  has 
been  performed  on  the  schedule,  the  contractor  recently 
conducted  a  high-level  Monte  Carlo  risk  analysis  on  two  major 
milestones  for  an  integrated  master  schedule  management 
review  meeting.  This  high-level  risk  analysis  shows  the 
probability  of  completing  the  key  milestones  on  time  and 
identifies  mitigating  actions  to  prevent  delays.  Program  officials 
stated  they  are  interested  in  periodic  risk  analysis  and  intend  to 
include  a  schedule  risk  analysis  requirement  in  the  contract 
within  the  next  few  months.  A  schedule  risk  analysis  can  be 
used  to  determine  a  level  of  confidence  in  meeting  the 
completion  date  or  whether  proper  reserves  have  been 
incorporated  into  the  schedule.  Such  an  analysis  will  calculate 
schedule  reserve,  which  can  be  set  aside  for  those  activities 
identified  as  high  risk.  Without  this  reserve,  the  program  faces 
the  risk  of  delays  to  the  scheduled  completion  date  if  any  delays 
were  to  occur  on  critical  path  activities. 

We  found  no  instances  of  illogical  dates,  such  as  actual  start  or 
actual  finish  dates  in  the  future.  We  found  1 1 2  instances 
(5  percent)  of  out-of-sequence  logic;  that  is,  actual  progress 
recorded  on  activities  that,  according  to  schedule  logic,  should 
not  have  started  yet.  This  is  a  common  occurrence  in 
scheduling,  as  actual  events  often  override  planned  logic. 
However,  a  large  number  of  out-of-sequence  activities  may 
indicate  that  the  schedule  is  not  being  thoroughly  updated  to 
reflect  reality  on  a  periodic  basis. 


Source:  GAO  analysis  based  on  data  provided  by  the  GCSS-Army  PMO. 
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This  appendix  provides  the  results  of  our  analysis  of  the  extent  to  which 
the  processes  and  methodologies  used  to  develop  and  maintain  the  four 
ERP  cost  estimates  meet  the  characteristics  of  high-quality  cost  estimates.1 
The  four  characteristics  of  high-quality  estimates  are  explained  and 
mapped  to  the  12  steps  of  such  estimates  in  table  16. 


Table  16:  The  12  Steps  of  High-Quality  Cost  Estimating,  Mapped  to  the  Steps  of  a  High-Quality  Cost  Estimate 


Characteristic  Explanation 


Step 


Well-documented 


The  documentation  should  address  the  purpose  of  the 
estimate,  the  program  background  and  system  description, 
its  schedule,  the  scope  of  the  estimate  (in  terms  of  time  and 
what  is  and  is  not  included),  the  ground  rules  and 
assumptions,  all  data  sources,  estimating  methodology  and 
rationale,  the  results  of  the  risk  analysis,  and  a  conclusion 
about  whether  the  cost  estimate  is  reasonable.  Therefore,  a 
good  cost  estimate — while  taking  the  form  of  a  single 
number — is  supported  by  detailed  documentation  that 
describes  how  it  was  derived  and  how  the  expected  funding 
will  be  spent  in  order  to  achieve  a  given  objective.  For 
example,  the  documentation  should  capture  in  writing  such 
things  as  the  source  data  used  and  their  significance,  the 
calculations  performed  and  their  results,  and  the  rationale  for 
choosing  a  particular  estimating  method  or  reference. 
Moreover,  this  information  should  be  captured  in  such  a  way 
that  the  data  used  to  derive  the  estimate  can  be  traced  back 
to  and  verified  against  their  sources,  allowing  for  the 
estimate  to  be  easily  replicated  and  updated.  Finally,  the 
cost  estimate  should  be  reviewed  and  accepted  by 
management  to  ensure  that  there  is  a  high  level  of 
confidence  in  the  estimating  process  and  the  estimate  itself. 


Step  1 :  Define  the  estimate’s  purpose,  scope,  and 
schedule 

Step  3:  Define  the  program  characteristics 

Step  5:  Identify  ground  rules  and  assumptions 

Step  6:  Obtain  the  data 

Step  10:  Document  the  estimate 

Step  1 1 :  Present  the  estimate  to  management  for 

approval 


Comprehensive 


The  cost  estimates  should  include  both  government  and 
contractor  costs  of  the  program  over  its  full  life  cycle,  from 
inception  of  the  program  through  design,  development, 
deployment,  and  operation  and  maintenance  to  retirement  of 
the  program.  They  should  also  completely  define  the 
program,  reflect  the  current  schedule,  and  be  technically 
reasonable.  Comprehensive  cost  estimates  should  provide  a 
level  of  detail  appropriate  to  ensure  that  cost  elements  are 
neither  omitted  nor  double  counted,  and  they  should 
document  all  cost-influencing  ground  rules  and  assumptions. 
Establishing  a  product-oriented  work  breakdown  structure 
(WBS)  is  a  best  practice  because  it  allows  a  program  to  track 
cost  and  schedule  by  defined  deliverables,  such  as  a 
hardware  or  software  component. 


Step  2:  Develop  the  estimating  plan 
Step  4:  Determine  the  estimating  structure 
Step  5:  Identify  ground  rules  and  assumptions3 


‘GAO-Og-SSP. 
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Characteristic 

Accurate 


Credible 


Explanation  Step 

The  cost  estimates  should  provide  for  results  that  are  Step  7:  Develop  the  point  estimateb 

unbiased,  and  they  should  not  be  overly  conservative  or  Step  1 2:  Update  the  estimate  to  reflect  actual  costs 

optimistic.  Estimates  are  accurate  when  they  are  based  on  ancj  changes 
an  assessment  of  most  likely  costs,  adjusted  properly  for 
inflation,  and  contain  few,  if  any,  minor  mistakes.  In  addition, 
the  estimates  should  be  updated  regularly  to  reflect  material 
changes  in  the  program,  such  as  when  schedules  or  other 
assumptions  change,  and  actual  costs  so  that  the  estimate  is 
always  reflecting  current  status.  Among  other  things,  the 
estimate  should  be  grounded  in  documented  assumptions 
and  a  historical  record  of  cost  estimating  and  actual 
experiences  on  other  comparable  programs. 

The  cost  estimates  should  discuss  any  limitations  of  the 
analysis  because  of  uncertainty  or  biases  surrounding  data 
or  assumptions.  Major  assumptions  should  be  varied,  and 
other  outcomes  recomputed  to  determine  how  sensitive  they 
are  to  changes  in  the  assumptions.  Risk  and  uncertainty 
analysis  should  be  performed  to  determine  the  level  of  risk 
associated  with  the  estimate.  Further,  the  estimate’s  results 
should  be  crosschecked,  and  an  independent  cost  estimate 
conducted  by  a  group  outside  the  acquiring  organization 
should  be  developed  to  determine  whether  other  estimating 
methods  produce  similar  results.  For  management  to  make 
good  decisions,  the  program  estimate  must  reflect  the 
degree  of  uncertainty,  so  that  a  level  of  confidence  can  be 
given  about  the  estimate.  Having  a  range  of  costs  around  a 
point  estimate  is  more  useful  to  decision  makers  because  it 
conveys  the  level  of  confidence  in  achieving  the  most  likely 
cost  and  also  informs  them  on  cost,  schedule,  and  technical 
risks. 


Step  7:  Compare  the  point  estimate  to  an 

independent  cost  estimate0 

Step  8:  Conduct  sensitivity  analysis 

Step  9:  Conduct  risk  and  uncertainty  analysis 


Source:  GAO-09-3SP. 

'This  step  applies  to  two  of  the  characteristics — well-documented  and  comprehensive. 
bA  point  estimate  is  a  single  cost  estimate  number  representing  the  most  likely  cost. 
'This  step  applies  to  two  of  the  characteristics — credible  and  accurate. 


Tables  17,  18,  19,  and  20  provide  the  detailed  results  of  our  analysis  of  the 
program  cost  estimates  for  DEAMS,  ECSS,  GFEBS  and  GCSS-Army.  “Not 
met”  means  the  program  provided  no  evidence  that  satisfies  any  of  the 
criterion.  “Minimally”  means  the  program  provided  evidence  that  satisfies 
a  small  portion  of  the  criterion.  “Partially”  means  the  program  provided 
evidence  that  satisfies  about  half  of  the  criterion.  “Substantially”  means 
the  program  provided  evidence  that  satisfies  a  large  portion  of  the 
criterion.  “Fully  met”  means  the  program  provided  evidence  that 
completely  satisfies  the  criterion. 
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Table  17:  Analysis  of  the  Air  Force’s  DEAMS  Cost  Estimate 

Four  characteristics  of  high- 

quality  cost  estimates 

Criterion  met 

Key  examples  of  rationale  for  assessment 

Well-documented 

Substantially 

The  purpose,  scope  and  schedule  of  the  estimate  were  clearly  defined.  Further,  the 
estimate  identified  all  the  ground  rules  and  assumptions  as  well  as  the  estimating 
methodology.  The  PMO  presented  evidence  of  receiving  approval  of  the  estimate 
through  briefings  to  management.  The  sources  of  data  the  estimate  was  based  on 
were  also  documented. 

However,  we  found  inconsistencies  when  comparing  the  program  requirements 
found  in  the  Cost  Analysis  Requirements  Document  (CARD)  with  the  requirements 
contained  in  the  cost  estimate.  For  example,  commercial-off-the-shelf  (COTS) 
software  licenses  requirements  outlined  in  the  CARD  do  not  match  the  assumptions 
used  in  the  cost  estimate. 

Comprehensive 

Fully  met 

The  program  provided  supporting  documentation  that  showed  the  ground  rules  and 
assumptions.  The  estimate  is  based  on  a  cost  element  structure  as  stated  in  the 
Department  of  Defense  Automated  Information  System  Economic  Analysis  Guide. 

The  program  also  provided  an  estimating  plan  that  included  the  cost  estimating 
schedule. 

Accurate 

Fully  met 

The  DEAMS  cost  model  details  the  calculations  and  inflation  indexes  underlying  the 
estimated  costs.  Calculations  within  the  model  can  be  traced  back  to  supporting 
documentation.  In  addition,  the  cost  model  is  updated  annually  to  incorporate  actual 
costs  expended  in  prior  fiscal  years. 

Credible 

Fully  met 

An  independent  cost  estimate  was  developed  by  the  Air  Force  Cost  Analysis 

Agency.  The  PMO  and  Air  Force  Cost  Analysis  Agency  also  conducted  analyses  to 
identify  the  cost  elements  with  the  greatest  degree  of  uncertainty,  and  determine  the 
cost  drivers  for  the  program,  and  performed  analyses  to  determine  the  impact  of 
changing  major  ground  rules  and  assumptions.  For  example,  during  the 
reconciliation  process,  there  was  debate  as  to  the  best  estimate  for  the  total  number 
of  DEAMS  users.  The  PMO  performed  a  sensitivity  analysis  on  this  parameter  to 
illustrate  the  total  life  cycle  cost  impact  of  changing  this  assumption. 

The  PMO  submitted  several  supporting  documents  that  detail  the  risk  and 
uncertainty  analysis  performed  on  the  cost  estimates.  In  addition  to  the  risk  and 
uncertainty  analysis,  the  PMO  implemented  a  risk  management  process  at  the 
inception  of  the  program  and  is  planned  to  continue  throughout  the  program’s  life. 

Source:  GAO  analysis  based  on  data  provided  by  the  DEAMS  PMO. 
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Appendix  V:  Assessments  of  Four  DOD  ERP 

Program  Cost  Estimates 

Table  18:  Analysis  of  the  Air  Force’s  ECSS  Cost  Estimate 

Four  characteristics  of  high- 

quality  cost  estimates 

Criterion  met 

Key  examples  of  rationale  for  assessment 

Well-documented 

Substantially 

The  purpose,  scope  and  schedule  of  the  estimate  were  clearly  defined.  The  PMO 
presented  evidence  of  receiving  approval  of  the  estimate  through  briefings  to 
management.  The  data  sources  were  also  documented.  The  PMO  also  provided 
ample  descriptions  of  the  methodology  used  to  derive  the  estimates.  However,  our 
analysis  found  inconsistencies  between  requirements  found  in  the  CARD  and 
assumptions  used  to  calculate  the  estimate.  For  example,  personnel  requirements 
and  the  number  of  reports,  interfaces,  conversions,  and  extensions  were  different 
between  the  two  documents. 

Comprehensive 

Fully  met 

The  program  provided  supporting  documentation  that  showed  the  ground  rules  and 
assumptions.  The  estimate  is  prepared  in  accordance  with  the  Office  of  the 

Secretary  of  Defense  ERP  work  breakdown  structure  as  stated  in  draft  DOD 
guidance.8  The  program  also  provided  an  estimating  plan  that  included  the  cost 
estimating  schedule. 

Accurate 

Substantially 

The  ECSS  cost  model  details  the  calculations  and  inflation  indexes  underlying  the 
estimated  costs.  Calculations  within  the  model  can  be  traced  back  to  supporting 
documentation.  However,  our  analysis  found  minor  inconsistencies  when  cross¬ 
checking  costs  that  were  presented  to  management  and  the  underlying  calculations 
within  the  model.  For  example,  estimates  for  data  migration,  data  cleansing,  and 
help  desk  within  the  cost  model  do  not  match  the  cost  estimates  presented  to 
management. 

ECSS  PMO  officials  stated  they  cannot  compare  actual  costs  to  the  cost  estimate 
because  they  do  not  yet  have  an  approved  baseline.  However,  these  officials  stated 
the  program  has  a  Baseline  Change  Board  that  holds  monthly  Resource  Board 
meetings  during  which  officials  review  the  program  baseline  to  assess  the  potential 
impacts  of  proposed  changes  to  all  aspects  of  the  program’s  life  cycle. 

Credible 

Partially 

An  independent  cost  estimate  was  created  by  the  Air  Force  Cost  Analysis  Agency. 
ECSS  PMO  officials  stated  that  the  cost  estimate  was  adjusted  based  on  sensitivity 
analyses.  However,  the  cost  estimate  model  does  not  include  evidence  of  a 
sensitivity  analysis.  Because  the  Air  Force  did  not  conduct  a  sensitivity  analysis  to 
identify  the  effects  of  uncertainties  associated  with  different  assumptions,  there  is  an 
increased  risk  that  decisions  will  be  made  without  a  clear  understanding  of  the 
possible  impact  on  cost  and  benefit  estimates. 

The  ECSS  PMO  performed  a  cost  risk  and  uncertainty  analysis.  This  analysis 
shows  that  the  service  cost  position  is  at  the  60  percent  confidence  level-meaning 
there  is  a  40  percent  chance  of  a  cost  overrun.  In  addition  to  the  risk  and  uncertainty 
analysis,  the  PMO  has  implemented  a  risk  management  process  to  identify  and 
mitigate  schedule,  cost,  and  performance  risks. 

Source:  GAO  analysis  based  on  data  provided  by  the  ECSS  PMO. 

aMIL-HDBK-881. 
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Appendix  V:  Assessments  of  Four  DOD  ERP 

Program  Cost  Estimates 

Table  19:  Analysis  of  the  Army’s  GFEBS  Cost  Estimate 

Four  characteristics  of  high- 

quality  cost  estimates 

Criterion  met 

Key  examples  of  rationale  for  assessment 

Well-documented 

Fully  met 

The  purpose,  scope  and  schedule  of  the  estimate  were  clearly  defined.  Further,  the 
documentation  identified  all  the  ground  rules  and  assumptions  as  well  as  the 
estimating  methodology.  The  PMO  presented  evidence  of  receiving  approval  of  the 
estimate  through  briefings  to  management.  The  sources  of  data  the  estimate  was 
based  on  were  also  documented. 

Comprehensive 

Fully  met 

The  program  provided  supporting  documentation  that  showed  the  ground  rules  and 
assumptions  underlying  the  cost  estimate.  The  estimate  is  based  on  a  cost 
estimating  structure  as  dictated  by  the  Department  of  the  Army  Economic  Analysis 
Manual.  The  program  also  provided  an  estimating  plan  that  included  the  cost 
estimating  schedule. 

Accurate 

Substantially 

The  GFEBS  cost  estimate  details  the  calculations  and  inflation  indexes  underlying 
the  estimated  costs.  Calculations  within  the  model  can  be  traced  back  to  supporting 
documentation.  In  addition,  evidence  was  provided  that  shows  how  estimated  costs 
were  derived  based  on  actual  costs  incurred  to  date.  For  example,  the  estimated 
cost  for  program  management  is  based  on  actual  historical  program  management 
costs.  However,  because  a  cost  uncertainty  analysis  has  not  been  performed,  DOD 
cannot  guarantee  that  the  estimate  represents  most  likely  costs  to  be  incurred. 

Credible 

Minimally 

An  independent  cost  estimate  was  created  by  the  Office  of  the  Deputy  Assistant 
Secretary  of  the  Army  for  Cost  and  Economics.  However,  the  estimate  does  not 
include  either  a  sensitivity  or  risk  and  uncertainty  analysis.  The  GFEBS  PMO  stated 
that  it  has  adequately  accounted  for  risks  in  the  cost  estimate  based  on  the  maturity 
of  the  program  and  the  reconciliation  process  between  the  PMO  estimate  and  the 
independent  cost  estimate.  However,  because  the  Army  did  not  conduct  a 
sensitivity  analysis  to  identify  the  effect  of  uncertainties  associated  with  different 
assumptions,  there  is  an  increased  risk  that  decisions  will  be  made  without  a  clear 
understanding  of  the  possible  impact  on  cost  and  benefit  estimates. 

Source:  GAO  analysis  based  on  data  provided  by  the  GFEBS  PMO. 
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Appendix  V:  Assessments  of  Four  DOD  ERP 
Program  Cost  Estimates 


Table  20:  Analysis  of  the  Army’s  GCSS-Army  Cost  Estimate 


Four  characteristics  of 
high-quality  cost 
estimates 

Criterion  met 

Key  examples  of  rationale  for  assessment 

Well-documented 

Substantially 

The  purpose,  scope  and  schedule  of  the  cost  estimate  were  clearly  defined.  The 
program  has  a  current  technical  baseline  document,  and  the  PMO  presented 
evidence  of  receiving  approval  of  the  estimate  through  briefings  to  management. 
However,  the  Economic  Analysis  documentation  describing  the  cost  estimate 
presents  costs  at  a  high  level  but  does  not  provide  details  on  lower  level  cost 
elements. 

Comprehensive 

Substantially 

The  GCSS-Army  PMO  uses  a  “hybrid”  work  breakdown  structure  for  the  program 
based  on  its  collaboration  with  the  Office  of  the  Secretary  of  Defense  Cost  and 
Resource  Center.  This  hybrid  work  breakdown  structure  while  not  entirely  product- 
oriented,  standardizes  the  vocabulary  for  cost  elements  for  automated  information 
systems.  Because  there  is  currently  no  standardized  work  breakdown  structure  in 
use  by  DOD  that  corresponds  to  the  implementation  of  an  ERP  system,  the  PMO 
worked  closely  with  the  Office  of  the  Secretary  of  Defense  Cost  and  Resource 

Center  to  develop  a  mutually  acceptable  work  breakdown  structure  that  meets  best 
practices. 

In  addition,  the  program  provided  supporting  documentation  that  showed  the  ground 
rules  and  assumptions  used  to  generate  the  cost  estimate.  However,  our  analysis 
shows  that  not  all  ground  rules  and  assumptions  were  used  to  develop  the  cost  risk 
and  uncertainty  analysis.  For  example,  there  are  several  assumptions  associated 
with  the  number  of  software  licenses,  yet  the  risk  and  uncertainty  analysis  does  not 
reflect  any  risk  associated  with  these  assumptions. 

Accurate 

Partially 

The  cost  estimate  model  shows  the  methodology  and  calculations  used  to  prepare 
the  estimate.  However,  because  the  PMO  did  not  provide  supporting  documentation 
that  details  the  use  of  actual  costs  to  derive  cost  estimates,  we  are  unable  to  verify 
the  quality  of  the  cost  estimates.  Programs  should  be  monitored  continuously  for 
their  cost-effectiveness  by  comparing  planned  and  actual  performance  against  the 
approved  program  baseline.  The  estimates  should  be  updated  with  actual  costs  so 
that  it  is  always  relevant  and  current.  This  results  in  a  higher-quality  cost  estimate 
and  provides  an  opportunity  to  incorporate  lessons  learned. 
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Appendix  V:  Assessments  of  Four  DOD  ERP 
Program  Cost  Estimates 


Four  characteristics  of 
high-quality  cost 
estimates 

Criterion  met 

Key  examples  of  rationale  for  assessment 

Credible 

Partially 

An  independent  cost  estimate  was  created  by  the  Army  Cost  Review  Board  Working 
Group.  However,  the  cost  estimate  does  not  include  a  sensitivity  analysis.  Because 
the  Army  did  not  conduct  a  sensitivity  analysis  to  identify  the  effects  of  uncertainties 
associated  with  different  assumptions,  there  is  an  increased  risk  that  decisions  will 
be  made  without  a  clear  understanding  of  the  possible  impact  on  cost  and  benefit 
estimates. 

The  supporting  documentation  shows  risk-adjusted  costs,  which  were  generated  by 
applying  probability  distributions  to  cost  elements  within  the  cost  model.  However, 
the  probability  distributions  applied  throughout  the  model  to  account  for  risks  are 
generalized  and  do  not  make  a  distinction  in  how  specific  risks  may  affect  specific 
cost  elements  differently. 

While  the  GCSS-Army  PMO  has  a  risk  process  to  identify,  analyze,  plan,  track, 
control,  and  communicate  risks,  our  analysis  found  that  the  PMO  did  not  adequately 
link  risks  to  the  cost  estimate.  For  example,  data  cleansing  and  data  migration  are 
noted  as  high-risks  within  the  risk  register  but  they  are  not  accounted  for  in  the  risk 
and  uncertainty  analysis.  Without  a  realistic  risk  and  uncertainty  analysis,  the  PMO 
can  neither  quantify  the  level  of  confidence  in  achieving  a  program  within  a  certain 
funding  level,  nor  determine  a  defensible  amount  of  contingency  reserve  to  quickly 
mitigate  risk. 

Source:  GAO  analysis  based  on  data  provided  by  the  GCSS-Army  PMO. 

Note:  The  focus  of  our  GCSS-Army  cost  assessment  is  the  “ratified”  GCSS-Army  Cost  Position  dated 
November  2006  because  the  ratified  Army  Cost  Position  represents  a  more  detailed  approach  to  the 
program’s  cost  estimating  process  compared  to  the  current  “federated”  approach  estimate  for 
Milestone  B.  The  current  federated  estimate,  which  reflects  the  federated  ERP  integration  strategy  for 
GCSS-Army  and  General  Funds  Enterprise  Business  Systems  (GFEBS),  was  developed  within  40 
days  as  mandated  by  the  Department  of  the  Army.  The  PMO  plans  to  implement  a  more  detailed  cost 
estimating  process  for  their  federated  Army  Cost  Position  in  preparation  for  Milestone  C  in  February 
2011. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
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